blob: 3b557fed0cf7f485781f04449ded8879e072630e [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreaufba74ea2018-12-22 11:19:45 +01004 version 2.0
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
William Lallemand142db372018-12-11 18:56:45 +0100339.4. Master CLI
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003410. Tricks for easier configuration management
3511. Well-known traps to avoid
3612. Debugging and performance issues
3713. Security considerations
38
39
401. Prerequisites
41----------------
42
43In this document it is assumed that the reader has sufficient administration
44skills on a UNIX-like operating system, uses the shell on a daily basis and is
45familiar with troubleshooting utilities such as strace and tcpdump.
46
47
482. Quick reminder about HAProxy's architecture
49----------------------------------------------
50
51HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is
52uses event multiplexing to schedule all of its activities instead of relying on
53the system to schedule between multiple activities. Most of the time it runs as
54a single process, so the output of "ps aux" on a system will report only one
55"haproxy" process, unless a soft reload is in progress and an older process is
56finishing its job in parallel to the new one. It is thus always easy to trace
57its activity using the strace utility.
58
59HAProxy is designed to isolate itself into a chroot jail during startup, where
60it cannot perform any file-system access at all. This is also true for the
61libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
62a running process will not be able to reload a configuration file to apply
63changes, instead a new process will be started using the updated configuration
64file. Some other less obvious effects are that some timezone files or resolver
65files the libc might attempt to access at run time will not be found, though
66this should generally not happen as they're not needed after startup. A nice
67consequence of this principle is that the HAProxy process is totally stateless,
68and no cleanup is needed after it's killed, so any killing method that works
69will do the right thing.
70
71HAProxy doesn't write log files, but it relies on the standard syslog protocol
72to send logs to a remote server (which is often located on the same system).
73
74HAProxy uses its internal clock to enforce timeouts, that is derived from the
75system's time but where unexpected drift is corrected. This is done by limiting
76the time spent waiting in poll() for an event, and measuring the time it really
77took. In practice it never waits more than one second. This explains why, when
78running strace over a completely idle process, periodic calls to poll() (or any
79of its variants) surrounded by two gettimeofday() calls are noticed. They are
80normal, completely harmless and so cheap that the load they imply is totally
81undetectable at the system scale, so there's nothing abnormal there. Example :
82
83 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
84 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
85 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
86 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
87 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
88 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
89
90HAProxy is a TCP proxy, not a router. It deals with established connections that
91have been validated by the kernel, and not with packets of any form nor with
92sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
93may prevent it from binding a port. It relies on the system to accept incoming
94connections and to initiate outgoing connections. An immediate effect of this is
95that there is no relation between packets observed on the two sides of a
96forwarded connection, which can be of different size, numbers and even family.
97Since a connection may only be accepted from a socket in LISTEN state, all the
98sockets it is listening to are necessarily visible using the "netstat" utility
99to show listening sockets. Example :
100
101 # netstat -ltnp
102 Active Internet connections (only servers)
103 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
104 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
105 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
106 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
107
108
1093. Starting HAProxy
110-------------------
111
112HAProxy is started by invoking the "haproxy" program with a number of arguments
113passed on the command line. The actual syntax is :
114
115 $ haproxy [<options>]*
116
117where [<options>]* is any number of options. An option always starts with '-'
118followed by one of more letters, and possibly followed by one or multiple extra
119arguments. Without any option, HAProxy displays the help page with a reminder
120about supported options. Available options may vary slightly based on the
121operating system. A fair number of these options overlap with an equivalent one
122if the "global" section. In this case, the command line always has precedence
123over the configuration file, so that the command line can be used to quickly
124enforce some settings without touching the configuration files. The current
125list of options is :
126
127 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200128 file/directory to be loaded and processed in the declaration order. It is
129 mostly useful when relying on the shell to load many files that are
130 numerically ordered. See also "-f". The difference between "--" and "-f" is
131 that one "-f" must be placed before each file name, while a single "--" is
132 needed before all file names. Both options can be used together, the
133 command line ordering still applies. When more than one file is specified,
134 each file must start on a section boundary, so the first keyword of each
135 file must be one of "global", "defaults", "peers", "listen", "frontend",
136 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200137
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200138 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
139 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400140 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200141 configuration files to be loaded ; only files with ".cfg" extension are
142 added, only non hidden files (not prefixed with ".") are added.
143 Configuration files are loaded and processed in their declaration order.
144 This option may be specified multiple times to load multiple files. See
145 also "--". The difference between "--" and "-f" is that one "-f" must be
146 placed before each file name, while a single "--" is needed before all file
147 names. Both options can be used together, the command line ordering still
148 applies. When more than one file is specified, each file must start on a
149 section boundary, so the first keyword of each file must be one of
150 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
151 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200152
153 -C <dir> : changes to directory <dir> before loading configuration
154 files. This is useful when using relative paths. Warning when using
155 wildcards after "--" which are in fact replaced by the shell before
156 starting haproxy.
157
158 -D : start as a daemon. The process detaches from the current terminal after
159 forking, and errors are not reported anymore in the terminal. It is
160 equivalent to the "daemon" keyword in the "global" section of the
161 configuration. It is recommended to always force it in any init script so
162 that a faulty configuration doesn't prevent the system from booting.
163
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200164 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200165 hostname. This is used only with peers replication. You can use the
166 variable $HAPROXY_LOCALPEER in the configuration file to reference the
167 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200168
169 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
170 builtin default value (usually 2000). Only useful for debugging.
171
172 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
173 "quiet".
174
William Lallemande202b1e2017-06-01 17:38:56 +0200175 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
176 the "global" section of the configuration. This mode will launch a "master"
177 which will monitor the "workers". Using this mode, you can reload HAProxy
178 directly by sending a SIGUSR2 signal to the master. The master-worker mode
179 is compatible either with the foreground or daemon mode. It is
180 recommended to use this mode with multiprocess and systemd.
181
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100182 -Ws : master-worker mode with support of `notify` type of systemd service.
183 This option is only available when HAProxy was built with `USE_SYSTEMD`
184 build option enabled.
185
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200186 -c : only performs a check of the configuration files and exits before trying
187 to bind. The exit status is zero if everything is OK, or non-zero if an
188 error is encountered.
189
190 -d : enable debug mode. This disables daemon mode, forces the process to stay
191 in foreground and to show incoming and outgoing events. It is equivalent to
192 the "global" section's "debug" keyword. It must never be used in an init
193 script.
194
195 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
196 can be used when suspecting that getaddrinfo() doesn't work as expected.
197 This option was made available because many bogus implementations of
198 getaddrinfo() exist on various systems and cause anomalies that are
199 difficult to troubleshoot.
200
Dan Lloyd8e48b872016-07-01 21:01:18 -0400201 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100202 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200203 <byte> before being passed to the caller. When <byte> is not specified, it
204 defaults to 0x50 ('P'). While this slightly slows down operations, it is
205 useful to reliably trigger issues resulting from missing initializations in
206 the code that cause random crashes. Note that -dM0 has the effect of
207 turning any malloc() into a calloc(). In any case if a bug appears or
208 disappears when using this option it means there is a bug in haproxy, so
209 please report it.
210
211 -dS : disable use of the splice() system call. It is equivalent to the
212 "global" section's "nosplice" keyword. This may be used when splice() is
213 suspected to behave improperly or to cause performance issues, or when
214 using strace to see the forwarded data (which do not appear when using
215 splice()).
216
217 -dV : disable SSL verify on the server side. It is equivalent to having
218 "ssl-server-verify none" in the "global" section. This is useful when
219 trying to reproduce production issues out of the production
220 environment. Never use this in an init script as it degrades SSL security
221 to the servers.
222
223 -db : disable background mode and multi-process mode. The process remains in
224 foreground. It is mainly used during development or during small tests, as
225 Ctrl-C is enough to stop the process. Never use it in an init script.
226
227 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
228 section's keyword "noepoll". It is mostly useful when suspecting a bug
229 related to this poller. On systems supporting epoll, the fallback will
230 generally be the "poll" poller.
231
232 -dk : disable the use of the "kqueue" poller. It is equivalent to the
233 "global" section's keyword "nokqueue". It is mostly useful when suspecting
234 a bug related to this poller. On systems supporting kqueue, the fallback
235 will generally be the "poll" poller.
236
237 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
238 section's keyword "nopoll". It is mostly useful when suspecting a bug
239 related to this poller. On systems supporting poll, the fallback will
240 generally be the "select" poller, which cannot be disabled and is limited
241 to 1024 file descriptors.
242
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100243 -dr : ignore server address resolution failures. It is very common when
244 validating a configuration out of production not to have access to the same
245 resolvers and to fail on server address resolution, making it difficult to
246 test a configuration. This option simply appends the "none" method to the
247 list of address resolution methods for all servers, ensuring that even if
248 the libc fails to resolve an address, the startup sequence is not
249 interrupted.
250
Willy Tarreau70060452015-12-14 12:46:07 +0100251 -m <limit> : limit the total allocatable memory to <limit> megabytes across
252 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200253 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100254 mostly used to force the processes to work in a constrained resource usage
255 scenario. It is important to note that the memory is not shared between
256 processes, so in a multi-process scenario, this value is first divided by
257 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200258
259 -n <limit> : limits the per-process connection limit to <limit>. This is
260 equivalent to the global section's keyword "maxconn". It has precedence
261 over this keyword. This may be used to quickly force lower limits to avoid
262 a service outage on systems where resource limits are too low.
263
264 -p <file> : write all processes' pids into <file> during startup. This is
265 equivalent to the "global" section's keyword "pidfile". The file is opened
266 before entering the chroot jail, and after doing the chdir() implied by
267 "-C". Each pid appears on its own line.
268
269 -q : set "quiet" mode. This disables some messages during the configuration
270 parsing and during startup. It can be used in combination with "-c" to
271 just check if a configuration file is valid or not.
272
William Lallemand142db372018-12-11 18:56:45 +0100273 -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
274 allows the access to every processes, running or leaving ones.
275 For security reasons, it is recommended to bind the master CLI to a local
276 UNIX socket. The bind options are the same as the keyword "bind" in
277 the configuration file with words separated by commas instead of spaces.
278
279 Note that this socket can't be used to retrieve the listening sockets from
280 an old process during a seamless reload.
281
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200282 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
283 completion to ask them to finish what they are doing and to leave. <pid>
284 is a list of pids to signal (one per argument). The list ends on any
285 option starting with a "-". It is not a problem if the list of pids is
286 empty, so that it can be built on the fly based on the result of a command
287 like "pidof" or "pgrep".
288
289 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
290 boot completion to terminate them immediately without finishing what they
291 were doing. <pid> is a list of pids to signal (one per argument). The list
292 is ends on any option starting with a "-". It is not a problem if the list
293 of pids is empty, so that it can be built on the fly based on the result of
294 a command like "pidof" or "pgrep".
295
296 -v : report the version and build date.
297
298 -vv : display the version, build options, libraries versions and usable
299 pollers. This output is systematically requested when filing a bug report.
300
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200301 -x <unix_socket> : connect to the specified socket and try to retrieve any
302 listening sockets from the old process, and use them instead of trying to
303 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200304 reloading the configuration on Linux. The capability must be enable on the
305 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200306
Dan Lloyd8e48b872016-07-01 21:01:18 -0400307A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200308mode, storing existing pids to a pid file and using this pid file to notify
309older processes to finish before leaving :
310
311 haproxy -f /etc/haproxy.cfg \
312 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
313
314When the configuration is split into a few specific files (eg: tcp vs http),
315it is recommended to use the "-f" option :
316
317 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
318 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
319 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
320 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
321
322When an unknown number of files is expected, such as customer-specific files,
323it is recommended to assign them a name starting with a fixed-size sequence
324number and to use "--" to load them, possibly after loading some defaults :
325
326 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
327 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
328 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
329 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
330 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
331
332Sometimes a failure to start may happen for whatever reason. Then it is
333important to verify if the version of HAProxy you are invoking is the expected
334version and if it supports the features you are expecting (eg: SSL, PCRE,
335compression, Lua, etc). This can be verified using "haproxy -vv". Some
336important information such as certain build options, the target system and
337the versions of the libraries being used are reported there. It is also what
338you will systematically be asked for when posting a bug report :
339
340 $ haproxy -vv
341 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
342 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
343
344 Build options :
345 TARGET = linux2628
346 CPU = generic
347 CC = gcc
348 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
349 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
350 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
351
352 Default settings :
353 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
354
355 Encrypted password support via crypt(3): yes
356 Built with zlib version : 1.2.6
357 Compression algorithms supported : identity("identity"), deflate("deflate"), \
358 raw-deflate("deflate"), gzip("gzip")
359 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
360 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
361 OpenSSL library supports TLS extensions : yes
362 OpenSSL library supports SNI : yes
363 OpenSSL library supports prefer-server-ciphers : yes
364 Built with PCRE version : 8.12 2011-01-15
365 PCRE library supports JIT : no (USE_PCRE_JIT not set)
366 Built with Lua version : Lua 5.3.1
367 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
368
369 Available polling systems :
370 epoll : pref=300, test result OK
371 poll : pref=200, test result OK
372 select : pref=150, test result OK
373 Total: 3 (3 usable), will use epoll.
374
375The relevant information that many non-developer users can verify here are :
376 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
377 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
378 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
379 fact "1.6-dev7". This is the 7th development version of what will become
380 version 1.6 in the future. A development version not suitable for use in
381 production (unless you know exactly what you are doing). A stable version
382 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
383 14th level of fix on top of version 1.5. This is a production-ready version.
384
385 - the release date : 2015/10/08. It is represented in the universal
386 year/month/day format. Here this means August 8th, 2015. Given that stable
387 releases are issued every few months (1-2 months at the beginning, sometimes
388 6 months once the product becomes very stable), if you're seeing an old date
389 here, it means you're probably affected by a number of bugs or security
390 issues that have since been fixed and that it might be worth checking on the
391 official site.
392
393 - build options : they are relevant to people who build their packages
394 themselves, they can explain why things are not behaving as expected. For
395 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400396 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200397 code optimization (-O0) so it will perform poorly in terms of performance.
398
399 - libraries versions : zlib version is reported as found in the library
400 itself. In general zlib is considered a very stable product and upgrades
401 are almost never needed. OpenSSL reports two versions, the version used at
402 build time and the one being used, as found on the system. These ones may
403 differ by the last letter but never by the numbers. The build date is also
404 reported because most OpenSSL bugs are security issues and need to be taken
405 seriously, so this library absolutely needs to be kept up to date. Seeing a
406 4-months old version here is highly suspicious and indeed an update was
407 missed. PCRE provides very fast regular expressions and is highly
408 recommended. Certain of its extensions such as JIT are not present in all
409 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400410 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200411 scripting language, HAProxy expects version 5.3 which is very young since
412 it was released a little time before HAProxy 1.6. It is important to check
413 on the Lua web site if some fixes are proposed for this branch.
414
415 - Available polling systems will affect the process's scalability when
416 dealing with more than about one thousand of concurrent connections. These
417 ones are only available when the correct system was indicated in the TARGET
418 variable during the build. The "epoll" mechanism is highly recommended on
419 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
420 will result in poll() or even select() being used, causing a high CPU usage
421 when dealing with a lot of connections.
422
423
4244. Stopping and restarting HAProxy
425----------------------------------
426
427HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
428SIGTERM signal is sent to the haproxy process, it immediately quits and all
429established connections are closed. The graceful stop is triggered when the
430SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
431from listening ports, but continue to process existing connections until they
432close. Once the last connection is closed, the process leaves.
433
434The hard stop method is used for the "stop" or "restart" actions of the service
435management script. The graceful stop is used for the "reload" action which
436tries to seamlessly reload a new configuration in a new process.
437
438Both of these signals may be sent by the new haproxy process itself during a
439reload or restart, so that they are sent at the latest possible moment and only
440if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
441(graceful) options respectively.
442
William Lallemande202b1e2017-06-01 17:38:56 +0200443In master-worker mode, it is not needed to start a new haproxy process in
444order to reload the configuration. The master process reacts to the SIGUSR2
445signal by reexecuting itself with the -sf parameter followed by the PIDs of
446the workers. The master will then parse the configuration file and fork new
447workers.
448
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200449To understand better how these signals are used, it is important to understand
450the whole restart mechanism.
451
452First, an existing haproxy process is running. The administrator uses a system
453specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
454take the new configuration file into effect. What happens then is the following.
455First, the service script (/etc/init.d/haproxy or equivalent) will verify that
456the configuration file parses correctly using "haproxy -c". After that it will
457try to start haproxy with this configuration file, using "-st" or "-sf".
458
459Then HAProxy tries to bind to all listening ports. If some fatal errors happen
460(eg: address not present on the system, permission denied), the process quits
461with an error. If a socket binding fails because a port is already in use, then
462the process will first send a SIGTTOU signal to all the pids specified in the
463"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
464all existing haproxy processes to temporarily stop listening to their ports so
465that the new process can try to bind again. During this time, the old process
466continues to process existing connections. If the binding still fails (because
467for example a port is shared with another daemon), then the new process sends a
468SIGTTIN signal to the old processes to instruct them to resume operations just
469as if nothing happened. The old processes will then restart listening to the
470ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400471dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200472
473If the new process manages to bind correctly to all ports, then it sends either
474the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
475of "-sf") to all processes to notify them that it is now in charge of operations
476and that the old processes will have to leave, either immediately or once they
477have finished their job.
478
479It is important to note that during this timeframe, there are two small windows
480of a few milliseconds each where it is possible that a few connection failures
481will be noticed during high loads. Typically observed failure rates are around
4821 failure during a reload operation every 10000 new connections per second,
483which means that a heavily loaded site running at 30000 new connections per
484second may see about 3 failed connection upon every reload. The two situations
485where this happens are :
486
487 - if the new process fails to bind due to the presence of the old process,
488 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
489 typically lasts about one millisecond for a few tens of frontends, and
490 during which some ports will not be bound to the old process and not yet
491 bound to the new one. HAProxy works around this on systems that support the
492 SO_REUSEPORT socket options, as it allows the new process to bind without
493 first asking the old one to unbind. Most BSD systems have been supporting
494 this almost forever. Linux has been supporting this in version 2.0 and
495 dropped it around 2.2, but some patches were floating around by then. It
496 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400497 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200498 is 3.9 or newer, or that relevant patches were backported to your kernel
499 (less likely).
500
501 - when the old processes close the listening ports, the kernel may not always
502 redistribute any pending connection that was remaining in the socket's
503 backlog. Under high loads, a SYN packet may happen just before the socket
504 is closed, and will lead to an RST packet being sent to the client. In some
505 critical environments where even one drop is not acceptable, these ones are
506 sometimes dealt with using firewall rules to block SYN packets during the
507 reload, forcing the client to retransmit. This is totally system-dependent,
508 as some systems might be able to visit other listening queues and avoid
509 this RST. A second case concerns the ACK from the client on a local socket
510 that was in SYN_RECV state just before the close. This ACK will lead to an
511 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400512 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200513 will work well if applied one second or so before restarting the process.
514
515For the vast majority of users, such drops will never ever happen since they
516don't have enough load to trigger the race conditions. And for most high traffic
517users, the failure rate is still fairly within the noise margin provided that at
518least SO_REUSEPORT is properly supported on their systems.
519
520
5215. File-descriptor limitations
522------------------------------
523
524In order to ensure that all incoming connections will successfully be served,
525HAProxy computes at load time the total number of file descriptors that will be
526needed during the process's life. A regular Unix process is generally granted
5271024 file descriptors by default, and a privileged process can raise this limit
528itself. This is one reason for starting HAProxy as root and letting it adjust
529the limit. The default limit of 1024 file descriptors roughly allow about 500
530concurrent connections to be processed. The computation is based on the global
531maxconn parameter which limits the total number of connections per process, the
532number of listeners, the number of servers which have a health check enabled,
533the agent checks, the peers, the loggers and possibly a few other technical
534requirements. A simple rough estimate of this number consists in simply
535doubling the maxconn value and adding a few tens to get the approximate number
536of file descriptors needed.
537
538Originally HAProxy did not know how to compute this value, and it was necessary
539to pass the value using the "ulimit-n" setting in the global section. This
540explains why even today a lot of configurations are seen with this setting
541present. Unfortunately it was often miscalculated resulting in connection
542failures when approaching maxconn instead of throttling incoming connection
543while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400544remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200545
546Raising the number of file descriptors to accept even moderate loads is
547mandatory but comes with some OS-specific adjustments. First, the select()
548polling system is limited to 1024 file descriptors. In fact on Linux it used
549to be capable of handling more but since certain OS ship with excessively
550restrictive SELinux policies forbidding the use of select() with more than
5511024 file descriptors, HAProxy now refuses to start in this case in order to
552avoid any issue at run time. On all supported operating systems, poll() is
553available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400554so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200555very slow when the number of file descriptors increases. While HAProxy does its
556best to limit this performance impact (eg: via the use of the internal file
557descriptor cache and batched processing), a good rule of thumb is that using
558poll() with more than a thousand concurrent connections will use a lot of CPU.
559
560For Linux systems base on kernels 2.6 and above, the epoll() system call will
561be used. It's a much more scalable mechanism relying on callbacks in the kernel
562that guarantee a constant wake up time regardless of the number of registered
563monitored file descriptors. It is automatically used where detected, provided
564that HAProxy had been built for one of the Linux flavors. Its presence and
565support can be verified using "haproxy -vv".
566
567For BSD systems which support it, kqueue() is available as an alternative. It
568is much faster than poll() and even slightly faster than epoll() thanks to its
569batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
570with Linux's epoll(), its support and availability are reported in the output
571of "haproxy -vv".
572
573Having a good poller is one thing, but it is mandatory that the process can
574reach the limits. When HAProxy starts, it immediately sets the new process's
575file descriptor limits and verifies if it succeeds. In case of failure, it
576reports it before forking so that the administrator can see the problem. As
577long as the process is started by as root, there should be no reason for this
578setting to fail. However, it can fail if the process is started by an
579unprivileged user. If there is a compelling reason for *not* starting haproxy
580as root (eg: started by end users, or by a per-application account), then the
581file descriptor limit can be raised by the system administrator for this
582specific user. The effectiveness of the setting can be verified by issuing
583"ulimit -n" from the user's command line. It should reflect the new limit.
584
585Warning: when an unprivileged user's limits are changed in this user's account,
586it is fairly common that these values are only considered when the user logs in
587and not at all in some scripts run at system boot time nor in crontabs. This is
588totally dependent on the operating system, keep in mind to check "ulimit -n"
589before starting haproxy when running this way. The general advice is never to
590start haproxy as an unprivileged user for production purposes. Another good
591reason is that it prevents haproxy from enabling some security protections.
592
593Once it is certain that the system will allow the haproxy process to use the
594requested number of file descriptors, two new system-specific limits may be
595encountered. The first one is the system-wide file descriptor limit, which is
596the total number of file descriptors opened on the system, covering all
597processes. When this limit is reached, accept() or socket() will typically
598return ENFILE. The second one is the per-process hard limit on the number of
599file descriptors, it prevents setrlimit() from being set higher. Both are very
600dependent on the operating system. On Linux, the system limit is set at boot
601based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
602And the per-process hard limit is set to 1048576 by default, but it can be
603changed using the "fs.nr_open" sysctl.
604
605File descriptor limitations may be observed on a running process when they are
606set too low. The strace utility will report that accept() and socket() return
607"-1 EMFILE" when the process's limits have been reached. In this case, simply
608raising the "ulimit-n" value (or removing it) will solve the problem. If these
609system calls return "-1 ENFILE" then it means that the kernel's limits have
610been reached and that something must be done on a system-wide parameter. These
611trouble must absolutely be addressed, as they result in high CPU usage (when
612accept() fails) and failed connections that are generally visible to the user.
613One solution also consists in lowering the global maxconn value to enforce
614serialization, and possibly to disable HTTP keep-alive to force connections
615to be released and reused faster.
616
617
6186. Memory management
619--------------------
620
621HAProxy uses a simple and fast pool-based memory management. Since it relies on
622a small number of different object types, it's much more efficient to pick new
623objects from a pool which already contains objects of the appropriate size than
624to call malloc() for each different size. The pools are organized as a stack or
625LIFO, so that newly allocated objects are taken from recently released objects
626still hot in the CPU caches. Pools of similar sizes are merged together, in
627order to limit memory fragmentation.
628
629By default, since the focus is set on performance, each released object is put
630back into the pool it came from, and allocated objects are never freed since
631they are expected to be reused very soon.
632
633On the CLI, it is possible to check how memory is being used in pools thanks to
634the "show pools" command :
635
636 > show pools
637 Dumping pools usage. Use SIGQUIT to flush them.
Willy Tarreau0a93b642018-10-16 07:58:39 +0200638 - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
639 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
640 - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
641 - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
642 - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
643 - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
644 - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
645 - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
646 - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
647 - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
648 - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
649 - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
650 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
651 - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
652 - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
653 - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
654 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
655 - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
656 - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
657 Total: 19 pools, 42296 bytes allocated, 34266 used.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200658
659The pool name is only indicative, it's the name of the first object type using
660this pool. The size in parenthesis is the object size for objects in this pool.
661Object sizes are always rounded up to the closest multiple of 16 bytes. The
662number of objects currently allocated and the equivalent number of bytes is
663reported so that it is easy to know which pool is responsible for the highest
664memory usage. The number of objects currently in use is reported as well in the
665"used" field. The difference between "allocated" and "used" corresponds to the
Willy Tarreau0a93b642018-10-16 07:58:39 +0200666objects that have been freed and are available for immediate use. The address
667at the end of the line is the pool's address, and the following number is the
668pool index when it exists, or is reported as -1 if no index was assigned.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200669
670It is possible to limit the amount of memory allocated per process using the
671"-m" command line option, followed by a number of megabytes. It covers all of
672the process's addressable space, so that includes memory used by some libraries
673as well as the stack, but it is a reliable limit when building a resource
674constrained system. It works the same way as "ulimit -v" on systems which have
675it, or "ulimit -d" for the other ones.
676
677If a memory allocation fails due to the memory limit being reached or because
678the system doesn't have any enough memory, then haproxy will first start to
679free all available objects from all pools before attempting to allocate memory
680again. This mechanism of releasing unused memory can be triggered by sending
681the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
682to the flush will also be reported to stderr when the process runs in
683foreground.
684
685During a reload operation, the process switched to the graceful stop state also
686automatically performs some flushes after releasing any connection so that all
687possible memory is released to save it for the new process.
688
689
6907. CPU usage
691------------
692
693HAProxy normally spends most of its time in the system and a smaller part in
694userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
695connection setups and closes per second at 100% CPU on a single core. When one
696core is saturated, typical figures are :
697 - 95% system, 5% user for long TCP connections or large HTTP objects
698 - 85% system and 15% user for short TCP connections or small HTTP objects in
699 close mode
700 - 70% system and 30% user for small HTTP objects in keep-alive mode
701
702The amount of rules processing and regular expressions will increase the user
703land part. The presence of firewall rules, connection tracking, complex routing
704tables in the system will instead increase the system part.
705
706On most systems, the CPU time observed during network transfers can be cut in 4
707parts :
708 - the interrupt part, which concerns all the processing performed upon I/O
709 receipt, before the target process is even known. Typically Rx packets are
710 accounted for in interrupt. On some systems such as Linux where interrupt
711 processing may be deferred to a dedicated thread, it can appear as softirq,
712 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
713 this load is generally defined by the hardware settings, though in the case
714 of softirq it is often possible to remap the processing to another CPU.
715 This interrupt part will often be perceived as parasitic since it's not
716 associated with any process, but it actually is some processing being done
717 to prepare the work for the process.
718
719 - the system part, which concerns all the processing done using kernel code
720 called from userland. System calls are accounted as system for example. All
721 synchronously delivered Tx packets will be accounted for as system time. If
722 some packets have to be deferred due to queues filling up, they may then be
723 processed in interrupt context later (eg: upon receipt of an ACK opening a
724 TCP window).
725
726 - the user part, which exclusively runs application code in userland. HAProxy
727 runs exclusively in this part, though it makes heavy use of system calls.
728 Rules processing, regular expressions, compression, encryption all add to
729 the user portion of CPU consumption.
730
731 - the idle part, which is what the CPU does when there is nothing to do. For
732 example HAProxy waits for an incoming connection, or waits for some data to
733 leave, meaning the system is waiting for an ACK from the client to push
734 these data.
735
736In practice regarding HAProxy's activity, it is in general reasonably accurate
737(but totally inexact) to consider that interrupt/softirq are caused by Rx
738processing in kernel drivers, that user-land is caused by layer 7 processing
739in HAProxy, and that system time is caused by network processing on the Tx
740path.
741
742Since HAProxy runs around an event loop, it waits for new events using poll()
743(or any alternative) and processes all these events as fast as possible before
744going back to poll() waiting for new events. It measures the time spent waiting
745in poll() compared to the time spent doing processing events. The ratio of
746polling time vs total time is called the "idle" time, it's the amount of time
747spent waiting for something to happen. This ratio is reported in the stats page
748on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
749the load is extremely low. When it's close to 0%, it means that there is
750constantly some activity. While it cannot be very accurate on an overloaded
751system due to other processes possibly preempting the CPU from the haproxy
752process, it still provides a good estimate about how HAProxy considers it is
753working : if the load is low and the idle ratio is low as well, it may indicate
754that HAProxy has a lot of work to do, possibly due to very expensive rules that
755have to be processed. Conversely, if HAProxy indicates the idle is close to
756100% while things are slow, it means that it cannot do anything to speed things
757up because it is already waiting for incoming data to process. In the example
758below, haproxy is completely idle :
759
760 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
761 Idle_pct: 100
762
763When the idle ratio starts to become very low, it is important to tune the
764system and place processes and interrupts correctly to save the most possible
765CPU resources for all tasks. If a firewall is present, it may be worth trying
766to disable it or to tune it to ensure it is not responsible for a large part
767of the performance limitation. It's worth noting that unloading a stateful
768firewall generally reduces both the amount of interrupt/softirq and of system
769usage since such firewalls act both on the Rx and the Tx paths. On Linux,
770unloading the nf_conntrack and ip_conntrack modules will show whether there is
771anything to gain. If so, then the module runs with default settings and you'll
772have to figure how to tune it for better performance. In general this consists
773in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
774disable the "pf" firewall and its stateful engine at the same time.
775
776If it is observed that a lot of time is spent in interrupt/softirq, it is
777important to ensure that they don't run on the same CPU. Most systems tend to
778pin the tasks on the CPU where they receive the network traffic because for
779certain workloads it improves things. But with heavily network-bound workloads
780it is the opposite as the haproxy process will have to fight against its kernel
781counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
782all sharing the same L3 cache tends to sensibly increase network performance
783because in practice the amount of work for haproxy and the network stack are
784quite close, so they can almost fill an entire CPU each. On Linux this is done
785using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
786interrupts are assigned under /proc/irq. Many network interfaces support
787multiple queues and multiple interrupts. In general it helps to spread them
788across a small number of CPU cores provided they all share the same L3 cache.
789Please always stop irq_balance which always does the worst possible thing on
790such workloads.
791
792For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
793compression, it may be worth using multiple processes dedicated to certain
794tasks, though there is no universal rule here and experimentation will have to
795be performed.
796
797In order to increase the CPU capacity, it is possible to make HAProxy run as
798several processes, using the "nbproc" directive in the global section. There
799are some limitations though :
800 - health checks are run per process, so the target servers will get as many
801 checks as there are running processes ;
802 - maxconn values and queues are per-process so the correct value must be set
803 to avoid overloading the servers ;
804 - outgoing connections should avoid using port ranges to avoid conflicts
805 - stick-tables are per process and are not shared between processes ;
806 - each peers section may only run on a single process at a time ;
807 - the CLI operations will only act on a single process at a time.
808
809With this in mind, it appears that the easiest setup often consists in having
810one first layer running on multiple processes and in charge for the heavy
811processing, passing the traffic to a second layer running in a single process.
812This mechanism is suited to SSL and compression which are the two CPU-heavy
813features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800814than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200815useful to pass client information to the next stage. When doing so, it is
816generally a good idea to bind all the single-process tasks to process number 1
817and extra tasks to next processes, as this will make it easier to generate
818similar configurations for different machines.
819
820On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
821more efficient when each process uses a distinct listening socket on the same
822IP:port ; this will make the kernel evenly distribute the load across all
823processes instead of waking them all up. Please check the "process" option of
824the "bind" keyword lines in the configuration manual for more information.
825
826
8278. Logging
828----------
829
830For logging, HAProxy always relies on a syslog server since it does not perform
831any file-system access. The standard way of using it is to send logs over UDP
832to the log server (by default on port 514). Very commonly this is configured to
833127.0.0.1 where the local syslog daemon is running, but it's also used over the
834network to log to a central server. The central server provides additional
835benefits especially in active-active scenarios where it is desirable to keep
836the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
837send its logs to the local syslog daemon, but it is not recommended at all,
838because if the syslog server is restarted while haproxy runs, the socket will
839be replaced and new logs will be lost. Since HAProxy will be isolated inside a
840chroot jail, it will not have the ability to reconnect to the new socket. It
841has also been observed in field that the log buffers in use on UNIX sockets are
842very small and lead to lost messages even at very light loads. But this can be
843fine for testing however.
844
845It is recommended to add the following directive to the "global" section to
846make HAProxy log to the local daemon using facility "local0" :
847
848 log 127.0.0.1:514 local0
849
850and then to add the following one to each "defaults" section or to each frontend
851and backend section :
852
853 log global
854
855This way, all logs will be centralized through the global definition of where
856the log server is.
857
858Some syslog daemons do not listen to UDP traffic by default, so depending on
859the daemon being used, the syntax to enable this will vary :
860
861 - on sysklogd, you need to pass argument "-r" on the daemon's command line
862 so that it listens to a UDP socket for "remote" logs ; note that there is
863 no way to limit it to address 127.0.0.1 so it will also receive logs from
864 remote systems ;
865
866 - on rsyslogd, the following lines must be added to the configuration file :
867
868 $ModLoad imudp
869 $UDPServerAddress *
870 $UDPServerRun 514
871
872 - on syslog-ng, a new source can be created the following way, it then needs
873 to be added as a valid source in one of the "log" directives :
874
875 source s_udp {
876 udp(ip(127.0.0.1) port(514));
877 };
878
879Please consult your syslog daemon's manual for more information. If no logs are
880seen in the system's log files, please consider the following tests :
881
882 - restart haproxy. Each frontend and backend logs one line indicating it's
883 starting. If these logs are received, it means logs are working.
884
885 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
886 activity that you expect to be logged. You should see the log messages
887 being sent using sendmsg() there. If they don't appear, restart using
888 strace on top of haproxy. If you still see no logs, it definitely means
889 that something is wrong in your configuration.
890
891 - run tcpdump to watch for port 514, for example on the loopback interface if
892 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
893 packets are seen there, it's the proof they're sent then the syslogd daemon
894 needs to be troubleshooted.
895
896While traffic logs are sent from the frontends (where the incoming connections
897are accepted), backends also need to be able to send logs in order to report a
898server state change consecutive to a health check. Please consult HAProxy's
899configuration manual for more information regarding all possible log settings.
900
Dan Lloyd8e48b872016-07-01 21:01:18 -0400901It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200902examples often suggest "local0" for traffic logs and "local1" for admin logs
903because they're never seen in field. A single facility would be enough as well.
904Having separate logs is convenient for log analysis, but it's also important to
905remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400906they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200907unauthorized people.
908
909For in-field troubleshooting without impacting the server's capacity too much,
910it is recommended to make use of the "halog" utility provided with HAProxy.
911This is sort of a grep-like utility designed to process HAProxy log files at
912a very fast data rate. Typical figures range between 1 and 2 GB of logs per
913second. It is capable of extracting only certain logs (eg: search for some
914classes of HTTP status codes, connection termination status, search by response
915time ranges, look for errors only), count lines, limit the output to a number
916of lines, and perform some more advanced statistics such as sorting servers
917by response time or error counts, sorting URLs by time or count, sorting client
918addresses by access count, and so on. It is pretty convenient to quickly spot
919anomalies such as a bot looping on the site, and block them.
920
921
9229. Statistics and monitoring
923----------------------------
924
Willy Tarreau44aed902015-10-13 14:45:29 +0200925It is possible to query HAProxy about its status. The most commonly used
926mechanism is the HTTP statistics page. This page also exposes an alternative
927CSV output format for monitoring tools. The same format is provided on the
928Unix socket.
929
930
9319.1. CSV format
932---------------
933
934The statistics may be consulted either from the unix socket or from the HTTP
935page. Both means provide a CSV format whose fields follow. The first line
936begins with a sharp ('#') and has one word per comma-delimited field which
937represents the title of the column. All other lines starting at the second one
938use a classical CSV format using a comma as the delimiter, and the double quote
939('"') as an optional text delimiter, but only if the enclosed text is ambiguous
940(if it contains a quote or a comma). The double-quote character ('"') in the
941text is doubled ('""'), which is the format that most tools recognize. Please
942do not insert any column before these ones in order not to break tools which
943use hard-coded column positions.
944
945In brackets after each field name are the types which may have a value for
946that field. The types are L (Listeners), F (Frontends), B (Backends), and
947S (Servers).
948
949 0. pxname [LFBS]: proxy name
950 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
951 any name for server/listener)
952 2. qcur [..BS]: current queued requests. For the backend this reports the
953 number queued without a server assigned.
954 3. qmax [..BS]: max value of qcur
955 4. scur [LFBS]: current sessions
956 5. smax [LFBS]: max sessions
957 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100958 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200959 8. bin [LFBS]: bytes in
960 9. bout [LFBS]: bytes out
961 10. dreq [LFB.]: requests denied because of security concerns.
962 - For tcp this is because of a matched tcp-request content rule.
963 - For http this is because of a matched http-request or tarpit rule.
964 11. dresp [LFBS]: responses denied because of security concerns.
965 - For http this is because of a matched http-request rule, or
966 "option checkcache".
967 12. ereq [LF..]: request errors. Some of the possible causes are:
968 - early termination from the client, before the request has been sent.
969 - read error from the client
970 - client timeout
971 - client closed connection
972 - various bad requests from the client.
973 - request was tarpitted.
974 13. econ [..BS]: number of requests that encountered an error trying to
975 connect to a backend server. The backend stat is the sum of the stat
976 for all servers of that backend, plus any connection errors not
977 associated with a particular server (such as the backend having no
978 active servers).
979 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
980 Some other errors are:
981 - write error on the client socket (won't be counted for the server stat)
982 - failure applying filters to the response.
983 15. wretr [..BS]: number of times a connection to a server was retried.
984 16. wredis [..BS]: number of times a request was redispatched to another
985 server. The server value counts the number of times that server was
986 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +0100987 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreau44aed902015-10-13 14:45:29 +0200988 18. weight [..BS]: total weight (backend), server weight (server)
989 19. act [..BS]: number of active servers (backend), server is active (server)
990 20. bck [..BS]: number of backup servers (backend), server is backup (server)
991 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
992 the server is up.)
993 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
994 transitions to the whole backend being down, rather than the sum of the
995 counters for each server.
996 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
997 24. downtime [..BS]: total downtime (in seconds). The value for the backend
998 is the downtime for the whole backend, not the sum of the server downtime.
999 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
1000 value is 0 (default, meaning no limit)
1001 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
1002 27. iid [LFBS]: unique proxy id
1003 28. sid [L..S]: server id (unique inside a proxy)
1004 29. throttle [...S]: current throttle percentage for the server, when
1005 slowstart is active, or no value if not in slowstart.
1006 30. lbtot [..BS]: total number of times a server was selected, either for new
1007 sessions, or when re-dispatching. The server counter is the number
1008 of times that server was selected.
1009 31. tracked [...S]: id of proxy/server if tracking is enabled.
1010 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
1011 33. rate [.FBS]: number of sessions per second over last elapsed second
1012 34. rate_lim [.F..]: configured limit on new sessions per second
1013 35. rate_max [.FBS]: max number of new sessions per second
1014 36. check_status [...S]: status of last health check, one of:
1015 UNK -> unknown
1016 INI -> initializing
1017 SOCKERR -> socket error
1018 L4OK -> check passed on layer 4, no upper layers testing enabled
1019 L4TOUT -> layer 1-4 timeout
1020 L4CON -> layer 1-4 connection problem, for example
1021 "Connection refused" (tcp rst) or "No route to host" (icmp)
1022 L6OK -> check passed on layer 6
1023 L6TOUT -> layer 6 (SSL) timeout
1024 L6RSP -> layer 6 invalid response - protocol error
1025 L7OK -> check passed on layer 7
1026 L7OKC -> check conditionally passed on layer 7, for example 404 with
1027 disable-on-404
1028 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1029 L7RSP -> layer 7 invalid response - protocol error
1030 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001031 Notice: If a check is currently running, the last known status will be
1032 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001033 37. check_code [...S]: layer5-7 code, if available
1034 38. check_duration [...S]: time in ms took to finish last health check
1035 39. hrsp_1xx [.FBS]: http responses with 1xx code
1036 40. hrsp_2xx [.FBS]: http responses with 2xx code
1037 41. hrsp_3xx [.FBS]: http responses with 3xx code
1038 42. hrsp_4xx [.FBS]: http responses with 4xx code
1039 43. hrsp_5xx [.FBS]: http responses with 5xx code
1040 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1041 45. hanafail [...S]: failed health checks details
1042 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1043 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001044 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001045 49. cli_abrt [..BS]: number of data transfers aborted by the client
1046 50. srv_abrt [..BS]: number of data transfers aborted by the server
1047 (inc. in eresp)
1048 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1049 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1050 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1051 (CPU/BW limit)
1052 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1053 55. lastsess [..BS]: number of seconds since last session assigned to
1054 server/backend
1055 56. last_chk [...S]: last health check contents or textual error
1056 57. last_agt [...S]: last agent check contents or textual error
1057 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1058 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1059 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1060 (0 for TCP)
1061 61. ttime [..BS]: the average total session time in ms over the 1024 last
1062 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001063 62. agent_status [...S]: status of last agent check, one of:
1064 UNK -> unknown
1065 INI -> initializing
1066 SOCKERR -> socket error
1067 L4OK -> check passed on layer 4, no upper layers testing enabled
1068 L4TOUT -> layer 1-4 timeout
1069 L4CON -> layer 1-4 connection problem, for example
1070 "Connection refused" (tcp rst) or "No route to host" (icmp)
1071 L7OK -> agent reported "up"
1072 L7STS -> agent reported "fail", "stop", or "down"
1073 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1074 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001075 65. check_desc [...S]: short human-readable description of check_status
1076 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001077 67. check_rise [...S]: server's "rise" parameter used by checks
1078 68. check_fall [...S]: server's "fall" parameter used by checks
1079 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1080 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1081 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1082 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001083 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001084 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001085 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001086 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001087 77: conn_rate [.F..]: number of connections over the last elapsed second
1088 78: conn_rate_max [.F..]: highest known conn_rate
1089 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001090 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001091 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001092 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001093 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Willy Tarreau44aed902015-10-13 14:45:29 +02001094
1095
Willy Tarreau5d8b9792016-03-11 11:09:34 +010010969.2) Typed output format
1097------------------------
1098
1099Both "show info" and "show stat" support a mode where each output value comes
1100with its type and sufficient information to know how the value is supposed to
1101be aggregated between processes and how it evolves.
1102
1103In all cases, the output consists in having a single value per line with all
1104the information split into fields delimited by colons (':').
1105
1106The first column designates the object or metric being dumped. Its format is
1107specific to the command producing this output and will not be described in this
1108section. Usually it will consist in a series of identifiers and field names.
1109
1110The second column contains 3 characters respectively indicating the origin, the
1111nature and the scope of the value being reported. The first character (the
1112origin) indicates where the value was extracted from. Possible characters are :
1113
1114 M The value is a metric. It is valid at one instant any may change depending
1115 on its nature .
1116
1117 S The value is a status. It represents a discrete value which by definition
1118 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1119 the PID of the process, etc.
1120
1121 K The value is a sorting key. It represents an identifier which may be used
1122 to group some values together because it is unique among its class. All
1123 internal identifiers are keys. Some names can be listed as keys if they
1124 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001125 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001126 most purposes keys may be considered as equivalent to configuration.
1127
1128 C The value comes from the configuration. Certain configuration values make
1129 sense on the output, for example a concurrent connection limit or a cookie
1130 name. By definition these values are the same in all processes started
1131 from the same configuration file.
1132
1133 P The value comes from the product itself. There are very few such values,
1134 most common use is to report the product name, version and release date.
1135 These elements are also the same between all processes.
1136
1137The second character (the nature) indicates the nature of the information
1138carried by the field in order to let an aggregator decide on what operation to
1139use to aggregate multiple values. Possible characters are :
1140
1141 A The value represents an age since a last event. This is a bit different
1142 from the duration in that an age is automatically computed based on the
1143 current date. A typical example is how long ago did the last session
1144 happen on a server. Ages are generally aggregated by taking the minimum
1145 value and do not need to be stored.
1146
1147 a The value represents an already averaged value. The average response times
1148 and server weights are of this nature. Averages can typically be averaged
1149 between processes.
1150
1151 C The value represents a cumulative counter. Such measures perpetually
1152 increase until they wrap around. Some monitoring protocols need to tell
1153 the difference between a counter and a gauge to report a different type.
1154 In general counters may simply be summed since they represent events or
1155 volumes. Examples of metrics of this nature are connection counts or byte
1156 counts.
1157
1158 D The value represents a duration for a status. There are a few usages of
1159 this, most of them include the time taken by the last health check and
1160 the time a server has spent down. Durations are generally not summed,
1161 most of the time the maximum will be retained to compute an SLA.
1162
1163 G The value represents a gauge. It's a measure at one instant. The memory
1164 usage or the current number of active connections are of this nature.
1165 Metrics of this type are typically summed during aggregation.
1166
1167 L The value represents a limit (generally a configured one). By nature,
1168 limits are harder to aggregate since they are specific to the point where
1169 they were retrieved. In certain situations they may be summed or be kept
1170 separate.
1171
1172 M The value represents a maximum. In general it will apply to a gauge and
1173 keep the highest known value. An example of such a metric could be the
1174 maximum amount of concurrent connections that was encountered in the
1175 product's life time. To correctly aggregate maxima, you are supposed to
1176 output a range going from the maximum of all maxima and the sum of all
1177 of them. There is indeed no way to know if they were encountered
1178 simultaneously or not.
1179
1180 m The value represents a minimum. In general it will apply to a gauge and
1181 keep the lowest known value. An example of such a metric could be the
1182 minimum amount of free memory pools that was encountered in the product's
1183 life time. To correctly aggregate minima, you are supposed to output a
1184 range going from the minimum of all minima and the sum of all of them.
1185 There is indeed no way to know if they were encountered simultaneously
1186 or not.
1187
1188 N The value represents a name, so it is a string. It is used to report
1189 proxy names, server names and cookie names. Names have configuration or
1190 keys as their origin and are supposed to be the same among all processes.
1191
1192 O The value represents a free text output. Outputs from various commands,
1193 returns from health checks, node descriptions are of such nature.
1194
1195 R The value represents an event rate. It's a measure at one instant. It is
1196 quite similar to a gauge except that the recipient knows that this measure
1197 moves slowly and may decide not to keep all values. An example of such a
1198 metric is the measured amount of connections per second. Metrics of this
1199 type are typically summed during aggregation.
1200
1201 T The value represents a date or time. A field emitting the current date
1202 would be of this type. The method to aggregate such information is left
1203 as an implementation choice. For now no field uses this type.
1204
1205The third character (the scope) indicates what extent the value reflects. Some
1206elements may be per process while others may be per configuration or per system.
1207The distinction is important to know whether or not a single value should be
1208kept during aggregation or if values have to be aggregated. The following
1209characters are currently supported :
1210
1211 C The value is valid for a whole cluster of nodes, which is the set of nodes
1212 communicating over the peers protocol. An example could be the amount of
1213 entries present in a stick table that is replicated with other peers. At
1214 the moment no metric use this scope.
1215
1216 P The value is valid only for the process reporting it. Most metrics use
1217 this scope.
1218
1219 S The value is valid for the whole service, which is the set of processes
1220 started together from the same configuration file. All metrics originating
1221 from the configuration use this scope. Some other metrics may use it as
1222 well for some shared resources (eg: shared SSL cache statistics).
1223
1224 s The value is valid for the whole system, such as the system's hostname,
1225 current date or resource usage. At the moment this scope is not used by
1226 any metric.
1227
1228Consumers of these information will generally have enough of these 3 characters
1229to determine how to accurately report aggregated information across multiple
1230processes.
1231
1232After this column, the third column indicates the type of the field, among "s32"
1233(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1234integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1235know the type before parsing the value in order to properly read it. For example
1236a string containing only digits is still a string an not an integer (eg: an
1237error code extracted by a check).
1238
1239Then the fourth column is the value itself, encoded according to its type.
1240Strings are dumped as-is immediately after the colon without any leading space.
1241If a string contains a colon, it will appear normally. This means that the
1242output should not be exclusively split around colons or some check outputs
1243or server addresses might be truncated.
1244
1245
12469.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001247-------------------------
1248
1249The stats socket is not enabled by default. In order to enable it, it is
1250necessary to add one line in the global section of the haproxy configuration.
1251A second line is recommended to set a larger timeout, always appreciated when
1252issuing commands by hand :
1253
1254 global
1255 stats socket /var/run/haproxy.sock mode 600 level admin
1256 stats timeout 2m
1257
1258It is also possible to add multiple instances of the stats socket by repeating
1259the line, and make them listen to a TCP port instead of a UNIX socket. This is
1260never done by default because this is dangerous, but can be handy in some
1261situations :
1262
1263 global
1264 stats socket /var/run/haproxy.sock mode 600 level admin
1265 stats socket ipv4@192.168.0.1:9999 level admin
1266 stats timeout 2m
1267
1268To access the socket, an external utility such as "socat" is required. Socat is
1269a swiss-army knife to connect anything to anything. We use it to connect
1270terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1271The two main syntaxes we'll use are the following :
1272
1273 # socat /var/run/haproxy.sock stdio
1274 # socat /var/run/haproxy.sock readline
1275
1276The first one is used with scripts. It is possible to send the output of a
1277script to haproxy, and pass haproxy's output to another script. That's useful
1278for retrieving counters or attack traces for example.
1279
1280The second one is only useful for issuing commands by hand. It has the benefit
1281that the terminal is handled by the readline library which supports line
1282editing and history, which is very convenient when issuing repeated commands
1283(eg: watch a counter).
1284
1285The socket supports two operation modes :
1286 - interactive
1287 - non-interactive
1288
1289The non-interactive mode is the default when socat connects to the socket. In
1290this mode, a single line may be sent. It is processed as a whole, responses are
1291sent back, and the connection closes after the end of the response. This is the
1292mode that scripts and monitoring tools use. It is possible to send multiple
1293commands in this mode, they need to be delimited by a semi-colon (';'). For
1294example :
1295
1296 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1297
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001298If a command needs to use a semi-colon or a backslash (eg: in a value), it
Joseph Herlant71b4b152018-11-13 16:55:16 -08001299must be preceded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001300
Willy Tarreau44aed902015-10-13 14:45:29 +02001301The interactive mode displays a prompt ('>') and waits for commands to be
1302entered on the line, then processes them, and displays the prompt again to wait
1303for a new command. This mode is entered via the "prompt" command which must be
1304sent on the first line in non-interactive mode. The mode is a flip switch, if
1305"prompt" is sent in interactive mode, it is disabled and the connection closes
1306after processing the last command of the same line.
1307
1308For this reason, when debugging by hand, it's quite common to start with the
1309"prompt" command :
1310
1311 # socat /var/run/haproxy readline
1312 prompt
1313 > show info
1314 ...
1315 >
1316
1317Since multiple commands may be issued at once, haproxy uses the empty line as a
1318delimiter to mark an end of output for each command, and takes care of ensuring
1319that no command can emit an empty line on output. A script can thus easily
1320parse the output even when multiple commands were pipelined on a single line.
1321
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001322Some commands may take an optional payload. To add one to a command, the first
1323line needs to end with the "<<\n" pattern. The next lines will be treated as
1324the payload and can contain as many lines as needed. To validate a command with
1325a payload, it needs to end with an empty line.
1326
1327Limitations do exist: the length of the whole buffer passed to the CLI must
1328not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1329last word of the line.
1330
1331When entering a paylod while in interactive mode, the prompt will change from
1332"> " to "+ ".
1333
Willy Tarreau44aed902015-10-13 14:45:29 +02001334It is important to understand that when multiple haproxy processes are started
1335on the same sockets, any process may pick up the request and will output its
1336own stats.
1337
1338The list of commands currently supported on the stats socket is provided below.
1339If an unknown command is sent, haproxy displays the usage message which reminds
1340all supported commands. Some commands support a more complex syntax, generally
1341it will explain what part of the command is invalid when this happens.
1342
Olivier Doucetd8703e82017-08-31 11:05:10 +02001343Some commands require a higher level of privilege to work. If you do not have
1344enough privilege, you will get an error "Permission denied". Please check
1345the "level" option of the "bind" keyword lines in the configuration manual
1346for more information.
1347
Willy Tarreau44aed902015-10-13 14:45:29 +02001348add acl <acl> <pattern>
1349 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1350 "show acl". This command does not verify if the entry already exists. This
1351 command cannot be used if the reference <acl> is a file also used with a map.
1352 In this case, you must use the command "add map" in place of "add acl".
1353
1354add map <map> <key> <value>
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001355add map <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001356 Add an entry into the map <map> to associate the value <value> to the key
1357 <key>. This command does not verify if the entry already exists. It is
1358 mainly used to fill a map after a clear operation. Note that if the reference
1359 <map> is a file and is shared with a map, this map will contain also a new
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001360 pattern entry. Using the payload syntax it is possible to add multiple
1361 key/value pairs by entering them on separate lines. On each new line, the
1362 first word is the key and the rest of the line is considered to be the value
1363 which can even contains spaces.
1364
1365 Example:
1366
1367 # socat /tmp/sock1 -
1368 prompt
1369
1370 > add map #-1 <<
1371 + key1 value1
1372 + key2 value2 with spaces
1373 + key3 value3 also with spaces
1374 + key4 value4
1375
1376 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001377
1378clear counters
1379 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001380 backend) and in each server. The accumulated counters are not affected. The
1381 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001382 can be used to get clean counters after an incident, without having to
1383 restart nor to clear traffic counters. This command is restricted and can
1384 only be issued on sockets configured for levels "operator" or "admin".
1385
1386clear counters all
1387 Clear all statistics counters in each proxy (frontend & backend) and in each
1388 server. This has the same effect as restarting. This command is restricted
1389 and can only be issued on sockets configured for level "admin".
1390
1391clear acl <acl>
1392 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1393 returned by "show acl". Note that if the reference <acl> is a file and is
1394 shared with a map, this map will be also cleared.
1395
1396clear map <map>
1397 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1398 returned by "show map". Note that if the reference <map> is a file and is
1399 shared with a acl, this acl will be also cleared.
1400
1401clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1402 Remove entries from the stick-table <table>.
1403
1404 This is typically used to unblock some users complaining they have been
1405 abusively denied access to a service, but this can also be used to clear some
1406 stickiness entries matching a server that is going to be replaced (see "show
1407 table" below for details). Note that sometimes, removal of an entry will be
1408 refused because it is currently tracked by a session. Retrying a few seconds
1409 later after the session ends is usual enough.
1410
1411 In the case where no options arguments are given all entries will be removed.
1412
1413 When the "data." form is used entries matching a filter applied using the
1414 stored data (see "stick-table" in section 4.2) are removed. A stored data
1415 type must be specified in <type>, and this data type must be stored in the
1416 table otherwise an error is reported. The data is compared according to
1417 <operator> with the 64-bit integer <value>. Operators are the same as with
1418 the ACLs :
1419
1420 - eq : match entries whose data is equal to this value
1421 - ne : match entries whose data is not equal to this value
1422 - le : match entries whose data is less than or equal to this value
1423 - ge : match entries whose data is greater than or equal to this value
1424 - lt : match entries whose data is less than this value
1425 - gt : match entries whose data is greater than this value
1426
1427 When the key form is used the entry <key> is removed. The key must be of the
1428 same type as the table, which currently is limited to IPv4, IPv6, integer and
1429 string.
1430
1431 Example :
1432 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1433 >>> # table: http_proxy, type: ip, size:204800, used:2
1434 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1435 bytes_out_rate(60000)=187
1436 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1437 bytes_out_rate(60000)=191
1438
1439 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1440
1441 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1442 >>> # table: http_proxy, type: ip, size:204800, used:1
1443 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1444 bytes_out_rate(60000)=191
1445 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1446 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1447 >>> # table: http_proxy, type: ip, size:204800, used:1
1448
1449del acl <acl> [<key>|#<ref>]
1450 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1451 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1452 this command delete only the listed reference. The reference can be found with
1453 listing the content of the acl. Note that if the reference <acl> is a file and
1454 is shared with a map, the entry will be also deleted in the map.
1455
1456del map <map> [<key>|#<ref>]
1457 Delete all the map entries from the map <map> corresponding to the key <key>.
1458 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1459 this command delete only the listed reference. The reference can be found with
1460 listing the content of the map. Note that if the reference <map> is a file and
1461 is shared with a acl, the entry will be also deleted in the map.
1462
1463disable agent <backend>/<server>
1464 Mark the auxiliary agent check as temporarily stopped.
1465
1466 In the case where an agent check is being run as a auxiliary check, due
1467 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001468 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001469 prevent any new agent checks from begin initiated until the agent
1470 re-enabled using enable agent.
1471
1472 When an agent is disabled the processing of an auxiliary agent check that
1473 was initiated while the agent was set as enabled is as follows: All
1474 results that would alter the weight, specifically "drain" or a weight
1475 returned by the agent, are ignored. The processing of agent check is
1476 otherwise unchanged.
1477
1478 The motivation for this feature is to allow the weight changing effects
1479 of the agent checks to be paused to allow the weight of a server to be
1480 configured using set weight without being overridden by the agent.
1481
1482 This command is restricted and can only be issued on sockets configured for
1483 level "admin".
1484
Olivier Houchard614f8d72017-03-14 20:08:46 +01001485disable dynamic-cookie backend <backend>
1486 Disable the generation of dynamic cookies fot the backend <backend>
1487
Willy Tarreau44aed902015-10-13 14:45:29 +02001488disable frontend <frontend>
1489 Mark the frontend as temporarily stopped. This corresponds to the mode which
1490 is used during a soft restart : the frontend releases the port but can be
1491 enabled again if needed. This should be used with care as some non-Linux OSes
1492 are unable to enable it back. This is intended to be used in environments
1493 where stopping a proxy is not even imaginable but a misconfigured proxy must
1494 be fixed. That way it's possible to release the port and bind it into another
1495 process to restore operations. The frontend will appear with status "STOP"
1496 on the stats page.
1497
1498 The frontend may be specified either by its name or by its numeric ID,
1499 prefixed with a sharp ('#').
1500
1501 This command is restricted and can only be issued on sockets configured for
1502 level "admin".
1503
1504disable health <backend>/<server>
1505 Mark the primary health check as temporarily stopped. This will disable
1506 sending of health checks, and the last health check result will be ignored.
1507 The server will be in unchecked state and considered UP unless an auxiliary
1508 agent check forces it down.
1509
1510 This command is restricted and can only be issued on sockets configured for
1511 level "admin".
1512
1513disable server <backend>/<server>
1514 Mark the server DOWN for maintenance. In this mode, no more checks will be
1515 performed on the server until it leaves maintenance.
1516 If the server is tracked by other servers, those servers will be set to DOWN
1517 during the maintenance.
1518
1519 In the statistics page, a server DOWN for maintenance will appear with a
1520 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1521
1522 Both the backend and the server may be specified either by their name or by
1523 their numeric ID, prefixed with a sharp ('#').
1524
1525 This command is restricted and can only be issued on sockets configured for
1526 level "admin".
1527
1528enable agent <backend>/<server>
1529 Resume auxiliary agent check that was temporarily stopped.
1530
1531 See "disable agent" for details of the effect of temporarily starting
1532 and stopping an auxiliary agent.
1533
1534 This command is restricted and can only be issued on sockets configured for
1535 level "admin".
1536
Olivier Houchard614f8d72017-03-14 20:08:46 +01001537enable dynamic-cookie backend <backend>
1538 Enable the generation of dynamic cookies fot the backend <backend>
1539 A secret key must also be provided
1540
Willy Tarreau44aed902015-10-13 14:45:29 +02001541enable frontend <frontend>
1542 Resume a frontend which was temporarily stopped. It is possible that some of
1543 the listening ports won't be able to bind anymore (eg: if another process
1544 took them since the 'disable frontend' operation). If this happens, an error
1545 is displayed. Some operating systems might not be able to resume a frontend
1546 which was disabled.
1547
1548 The frontend may be specified either by its name or by its numeric ID,
1549 prefixed with a sharp ('#').
1550
1551 This command is restricted and can only be issued on sockets configured for
1552 level "admin".
1553
1554enable health <backend>/<server>
1555 Resume a primary health check that was temporarily stopped. This will enable
1556 sending of health checks again. Please see "disable health" for details.
1557
1558 This command is restricted and can only be issued on sockets configured for
1559 level "admin".
1560
1561enable server <backend>/<server>
1562 If the server was previously marked as DOWN for maintenance, this marks the
1563 server UP and checks are re-enabled.
1564
1565 Both the backend and the server may be specified either by their name or by
1566 their numeric ID, prefixed with a sharp ('#').
1567
1568 This command is restricted and can only be issued on sockets configured for
1569 level "admin".
1570
1571get map <map> <value>
1572get acl <acl> <value>
1573 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1574 are the #<id> or the <file> returned by "show map" or "show acl". This command
1575 returns all the matching patterns associated with this map. This is useful for
1576 debugging maps and ACLs. The output format is composed by one line par
1577 matching type. Each line is composed by space-delimited series of words.
1578
1579 The first two words are:
1580
1581 <match method>: The match method applied. It can be "found", "bool",
1582 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1583 "dom", "end" or "reg".
1584
1585 <match result>: The result. Can be "match" or "no-match".
1586
1587 The following words are returned only if the pattern matches an entry.
1588
1589 <index type>: "tree" or "list". The internal lookup algorithm.
1590
1591 <case>: "case-insensitive" or "case-sensitive". The
1592 interpretation of the case.
1593
1594 <entry matched>: match="<entry>". Return the matched pattern. It is
1595 useful with regular expressions.
1596
1597 The two last word are used to show the returned value and its type. With the
1598 "acl" case, the pattern doesn't exist.
1599
1600 return=nothing: No return because there are no "map".
1601 return="<value>": The value returned in the string format.
1602 return=cannot-display: The value cannot be converted as string.
1603
1604 type="<type>": The type of the returned sample.
1605
1606get weight <backend>/<server>
1607 Report the current weight and the initial weight of server <server> in
1608 backend <backend> or an error if either doesn't exist. The initial weight is
1609 the one that appears in the configuration file. Both are normally equal
1610 unless the current weight has been changed. Both the backend and the server
1611 may be specified either by their name or by their numeric ID, prefixed with a
1612 sharp ('#').
1613
1614help
1615 Print the list of known keywords and their basic usage. The same help screen
1616 is also displayed for unknown commands.
1617
1618prompt
1619 Toggle the prompt at the beginning of the line and enter or leave interactive
1620 mode. In interactive mode, the connection is not closed after a command
1621 completes. Instead, the prompt will appear again, indicating the user that
1622 the interpreter is waiting for a new command. The prompt consists in a right
1623 angle bracket followed by a space "> ". This mode is particularly convenient
1624 when one wants to periodically check information such as stats or errors.
1625 It is also a good idea to enter interactive mode before issuing a "help"
1626 command.
1627
1628quit
1629 Close the connection when in interactive mode.
1630
Olivier Houchard614f8d72017-03-14 20:08:46 +01001631set dynamic-cookie-key backend <backend> <value>
1632 Modify the secret key used to generate the dynamic persistent cookies.
1633 This will break the existing sessions.
1634
Willy Tarreau44aed902015-10-13 14:45:29 +02001635set map <map> [<key>|#<ref>] <value>
1636 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1637 #<id> or <file> returned by "show map". If the <ref> is used in place of
1638 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1639
1640set maxconn frontend <frontend> <value>
1641 Dynamically change the specified frontend's maxconn setting. Any positive
1642 value is allowed including zero, but setting values larger than the global
1643 maxconn does not make much sense. If the limit is increased and connections
1644 were pending, they will immediately be accepted. If it is lowered to a value
1645 below the current number of connections, new connections acceptation will be
1646 delayed until the threshold is reached. The frontend might be specified by
1647 either its name or its numeric ID prefixed with a sharp ('#').
1648
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001649set maxconn server <backend/server> <value>
1650 Dynamically change the specified server's maxconn setting. Any positive
1651 value is allowed including zero, but setting values larger than the global
1652 maxconn does not make much sense.
1653
Willy Tarreau44aed902015-10-13 14:45:29 +02001654set maxconn global <maxconn>
1655 Dynamically change the global maxconn setting within the range defined by the
1656 initial global maxconn setting. If it is increased and connections were
1657 pending, they will immediately be accepted. If it is lowered to a value below
1658 the current number of connections, new connections acceptation will be
1659 delayed until the threshold is reached. A value of zero restores the initial
1660 setting.
1661
Willy Tarreau75c62c22018-11-22 11:02:09 +01001662set profiling { tasks } { on | off }
1663 Enables or disables CPU profiling for the indicated subsystem. This is
1664 equivalent to setting or clearing the "profiling" settings in the "global"
1665 section of the configuration file. Please also see "show profiling".
1666
Willy Tarreau44aed902015-10-13 14:45:29 +02001667set rate-limit connections global <value>
1668 Change the process-wide connection rate limit, which is set by the global
1669 'maxconnrate' setting. A value of zero disables the limitation. This limit
1670 applies to all frontends and the change has an immediate effect. The value
1671 is passed in number of connections per second.
1672
1673set rate-limit http-compression global <value>
1674 Change the maximum input compression rate, which is set by the global
1675 'maxcomprate' setting. A value of zero disables the limitation. The value is
1676 passed in number of kilobytes per second. The value is available in the "show
1677 info" on the line "CompressBpsRateLim" in bytes.
1678
1679set rate-limit sessions global <value>
1680 Change the process-wide session rate limit, which is set by the global
1681 'maxsessrate' setting. A value of zero disables the limitation. This limit
1682 applies to all frontends and the change has an immediate effect. The value
1683 is passed in number of sessions per second.
1684
1685set rate-limit ssl-sessions global <value>
1686 Change the process-wide SSL session rate limit, which is set by the global
1687 'maxsslrate' setting. A value of zero disables the limitation. This limit
1688 applies to all frontends and the change has an immediate effect. The value
1689 is passed in number of sessions per second sent to the SSL stack. It applies
1690 before the handshake in order to protect the stack against handshake abuses.
1691
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001692set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001693 Replace the current IP address of a server by the one provided.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001694 Optionnaly, the port can be changed using the 'port' parameter.
1695 Note that changing the port also support switching from/to port mapping
1696 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001697
1698set server <backend>/<server> agent [ up | down ]
1699 Force a server's agent to a new state. This can be useful to immediately
1700 switch a server's state regardless of some slow agent checks for example.
1701 Note that the change is propagated to tracking servers if any.
1702
Misiek43972902017-01-09 09:53:06 +01001703set server <backend>/<server> agent-addr <addr>
1704 Change addr for servers agent checks. Allows to migrate agent-checks to
1705 another address at runtime. You can specify both IP and hostname, it will be
1706 resolved.
1707
1708set server <backend>/<server> agent-send <value>
1709 Change agent string sent to agent check target. Allows to update string while
1710 changing server address to keep those two matching.
1711
Willy Tarreau44aed902015-10-13 14:45:29 +02001712set server <backend>/<server> health [ up | stopping | down ]
1713 Force a server's health to a new state. This can be useful to immediately
1714 switch a server's state regardless of some slow health checks for example.
1715 Note that the change is propagated to tracking servers if any.
1716
Baptiste Assmann50946562016-08-31 23:26:29 +02001717set server <backend>/<server> check-port <port>
1718 Change the port used for health checking to <port>
1719
Willy Tarreau44aed902015-10-13 14:45:29 +02001720set server <backend>/<server> state [ ready | drain | maint ]
1721 Force a server's administrative state to a new state. This can be useful to
1722 disable load balancing and/or any traffic to a server. Setting the state to
1723 "ready" puts the server in normal mode, and the command is the equivalent of
1724 the "enable server" command. Setting the state to "maint" disables any traffic
1725 to the server as well as any health checks. This is the equivalent of the
1726 "disable server" command. Setting the mode to "drain" only removes the server
1727 from load balancing but still allows it to be checked and to accept new
1728 persistent connections. Changes are propagated to tracking servers if any.
1729
1730set server <backend>/<server> weight <weight>[%]
1731 Change a server's weight to the value passed in argument. This is the exact
1732 equivalent of the "set weight" command below.
1733
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001734set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02001735 Change a server's FQDN to the value passed in argument. This requires the
1736 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001737
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02001738set severity-output [ none | number | string ]
1739 Change the severity output format of the stats socket connected to for the
1740 duration of the current session.
1741
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001742set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001743 This command is used to update an OCSP Response for a certificate (see "crt"
1744 on "bind" lines). Same controls are performed as during the initial loading of
1745 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02001746 DER encoded response from the OCSP server. This command is not supported with
1747 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02001748
1749 Example:
1750 openssl ocsp -issuer issuer.pem -cert server.pem \
1751 -host ocsp.issuer.com:80 -respout resp.der
1752 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1753 socat stdio /var/run/haproxy.stat
1754
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001755 using the payload syntax:
1756 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
1757 socat stdio /var/run/haproxy.stat
1758
Willy Tarreau44aed902015-10-13 14:45:29 +02001759set ssl tls-key <id> <tlskey>
1760 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1761 ultimate key, while the penultimate one is used for encryption (others just
1762 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1763 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
1764 bit TLS ticket key (ex. openssl rand -base64 48).
1765
1766set table <table> key <key> [data.<data_type> <value>]*
1767 Create or update a stick-table entry in the table. If the key is not present,
1768 an entry is inserted. See stick-table in section 4.2 to find all possible
1769 values for <data_type>. The most likely use consists in dynamically entering
1770 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1771 IP address or affect its quality of service. It is possible to pass multiple
1772 data_types in a single call.
1773
1774set timeout cli <delay>
1775 Change the CLI interface timeout for current connection. This can be useful
1776 during long debugging sessions where the user needs to constantly inspect
1777 some indicators without being disconnected. The delay is passed in seconds.
1778
1779set weight <backend>/<server> <weight>[%]
1780 Change a server's weight to the value passed in argument. If the value ends
1781 with the '%' sign, then the new weight will be relative to the initially
1782 configured weight. Absolute weights are permitted between 0 and 256.
1783 Relative weights must be positive with the resulting absolute weight is
1784 capped at 256. Servers which are part of a farm running a static
1785 load-balancing algorithm have stricter limitations because the weight
1786 cannot change once set. Thus for these servers, the only accepted values
1787 are 0 and 100% (or 0 and the initial weight). Changes take effect
1788 immediately, though certain LB algorithms require a certain amount of
1789 requests to consider changes. A typical usage of this command is to
1790 disable a server during an update by setting its weight to zero, then to
1791 enable it again after the update by setting it back to 100%. This command
1792 is restricted and can only be issued on sockets configured for level
1793 "admin". Both the backend and the server may be specified either by their
1794 name or by their numeric ID, prefixed with a sharp ('#').
1795
Willy Tarreaud6129fc2017-07-28 16:52:23 +02001796show acl [<acl>]
1797 Dump info about acl converters. Without argument, the list of all available
1798 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1799 the #<id> or <file>. The dump format is the same than the map even for the
1800 sample value. The data returned are not a list of available ACL, but are the
1801 list of all patterns composing any ACL. Many of these patterns can be shared
1802 with maps.
1803
1804show backend
1805 Dump the list of backends available in the running process
1806
William Lallemand67a234f2018-12-13 09:05:45 +01001807show cli level
1808 Display the CLI level of the current CLI session. The result could be
1809 'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
1810
1811 Example :
1812
1813 $ socat /tmp/sock1 readline
1814 prompt
1815 > operator
1816 > show cli level
1817 operator
1818 > user
1819 > show cli level
1820 user
1821 > operator
1822 Permission denied
1823
1824operator
1825 Decrease the CLI level of the current CLI session to operator. It can't be
1826 increase. See also "show cli level"
1827
1828user
1829 Decrease the CLI level of the current CLI session to user. It can't be
1830 increase. See also "show cli level"
1831
William Lallemand51132162016-12-16 16:38:58 +01001832show cli sockets
1833 List CLI sockets. The output format is composed of 3 fields separated by
1834 spaces. The first field is the socket address, it can be a unix socket, a
1835 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1836 The second field describe the level of the socket: 'admin', 'user' or
1837 'operator'. The last field list the processes on which the socket is bound,
1838 separated by commas, it can be numbers or 'all'.
1839
1840 Example :
1841
1842 $ echo 'show cli sockets' | socat stdio /tmp/sock1
1843 # socket lvl processes
1844 /tmp/sock1 admin all
1845 127.0.0.1:9999 user 2,3,4
1846 127.0.0.2:9969 user 2
1847 [::1]:9999 operator 2
1848
William Lallemand86d0df02017-11-24 21:36:45 +01001849show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01001850 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01001851
1852 $ echo 'show cache' | socat stdio /tmp/sock1
1853 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
1854 1 2 3 4
1855
1856 1. pointer to the cache structure
1857 2. cache name
1858 3. pointer to the mmap area (shctx)
1859 4. number of blocks available for reuse in the shctx
1860
1861 0x7f6ac6c5b4cc hash:286881868 size:39114 (39 blocks), refcount:9, expire:237
1862 1 2 3 4 5 6
1863
1864 1. pointer to the cache entry
1865 2. first 32 bits of the hash
1866 3. size of the object in bytes
1867 4. number of blocks used for the object
1868 5. number of transactions using the entry
1869 6. expiration time, can be negative if already expired
1870
Willy Tarreauae795722016-02-16 11:27:28 +01001871show env [<name>]
1872 Dump one or all environment variables known by the process. Without any
1873 argument, all variables are dumped. With an argument, only the specified
1874 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1875 Variables are dumped in the same format as they are stored or returned by the
1876 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1877 certain configuration files making heavy use of environment variables to
1878 ensure that they contain the expected values. This command is restricted and
1879 can only be issued on sockets configured for levels "operator" or "admin".
1880
Willy Tarreau35069f82016-11-25 09:16:37 +01001881show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02001882 Dump last known request and response errors collected by frontends and
1883 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01001884 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
1885 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01001886 will be used as the filter. If "request" or "response" is added after the
1887 proxy name or ID, only request or response errors will be dumped. This
1888 command is restricted and can only be issued on sockets configured for
1889 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02001890
1891 The errors which may be collected are the last request and response errors
1892 caused by protocol violations, often due to invalid characters in header
1893 names. The report precisely indicates what exact character violated the
1894 protocol. Other important information such as the exact date the error was
1895 detected, frontend and backend names, the server name (when known), the
1896 internal session ID and the source address which has initiated the session
1897 are reported too.
1898
1899 All characters are returned, and non-printable characters are encoded. The
1900 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1901 letter following a backslash. The backslash itself is encoded as '\\' to
1902 avoid confusion. Other non-printable characters are encoded '\xNN' where
1903 NN is the two-digits hexadecimal representation of the character's ASCII
1904 code.
1905
1906 Lines are prefixed with the position of their first character, starting at 0
1907 for the beginning of the buffer. At most one input line is printed per line,
1908 and large lines will be broken into multiple consecutive output lines so that
1909 the output never goes beyond 79 characters wide. It is easy to detect if a
1910 line was broken, because it will not end with '\n' and the next line's offset
1911 will be followed by a '+' sign, indicating it is a continuation of previous
1912 line.
1913
1914 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01001915 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02001916 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1917 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1918 response length 213 bytes, error at position 23:
1919
1920 00000 HTTP/1.0 200 OK\r\n
1921 00017 header/bizarre:blah\r\n
1922 00038 Location: blah\r\n
1923 00054 Long-line: this is a very long line which should b
1924 00104+ e broken into multiple lines on the output buffer,
1925 00154+ otherwise it would be too large to print in a ter
1926 00204+ minal\r\n
1927 00211 \r\n
1928
1929 In the example above, we see that the backend "http-in" which has internal
1930 ID 2 has blocked an invalid response from its server s2 which has internal
1931 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1932 received by frontend fe-eth0 whose ID is 1. The total response length was
1933 213 bytes when the error was detected, and the error was at byte 23. This
1934 is the slash ('/') in header name "header/bizarre", which is not a valid
1935 HTTP character for a header name.
1936
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02001937show fd [<fd>]
1938 Dump the list of either all open file descriptors or just the one number <fd>
1939 if specified. This is only aimed at developers who need to observe internal
1940 states in order to debug complex issues such as abnormal CPU usages. One fd
1941 is reported per lines, and for each of them, its state in the poller using
1942 upper case letters for enabled flags and lower case for disabled flags, using
1943 "P" for "polled", "R" for "ready", "A" for "active", the events status using
1944 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
1945 "I" for "input", a few other flags like "N" for "new" (just added into the fd
1946 cache), "U" for "updated" (received an update in the fd cache), "L" for
1947 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
1948 to the internal owner, the pointer to the I/O callback and its name when
1949 known. When the owner is a connection, the connection flags, and the target
1950 are reported (frontend, proxy or server). When the owner is a listener, the
1951 listener's state and its frontend are reported. There is no point in using
1952 this command without a good knowledge of the internals. It's worth noting
1953 that the output format may evolve over time so this output must not be parsed
1954 by tools designed to be durable.
1955
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001956show activity
1957 Reports some counters about internal events that will help developers and
1958 more generally people who know haproxy well enough to narrow down the causes
1959 of reports of abnormal behaviours. A typical example would be a properly
1960 running process never sleeping and eating 100% of the CPU. The output fields
1961 will be made of one line per metric, and per-thread counters on the same
1962 line. These counters are 32-bit and will wrap during the process' life, which
1963 is not a problem since calls to this command will typically be performed
1964 twice. The fields are purposely not documented so that their exact meaning is
1965 verified in the code where the counters are fed. These values are also reset
1966 by the "clear counters" command.
1967
Simon Horman05ee2132017-01-04 09:37:25 +01001968show info [typed|json]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001969 Dump info about haproxy status on current process. If "typed" is passed as an
1970 optional argument, field numbers, names and types are emitted as well so that
1971 external monitoring products can easily retrieve, possibly aggregate, then
1972 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01001973 its own line. If "json" is passed as an optional argument then
1974 information provided by "typed" output is provided in JSON format as a
1975 list of JSON objects. By default, the format contains only two columns
1976 delimited by a colon (':'). The left one is the field name and the right
1977 one is the value. It is very important to note that in typed output
1978 format, the dump for a single object is contiguous so that there is no
1979 need for a consumer to store everything at once.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001980
1981 When using the typed output format, each line is made of 4 columns delimited
1982 by colons (':'). The first column is a dot-delimited series of 3 elements. The
1983 first element is the numeric position of the field in the list (starting at
1984 zero). This position shall not change over time, but holes are to be expected,
1985 depending on build options or if some fields are deleted in the future. The
1986 second element is the field name as it appears in the default "show info"
1987 output. The third element is the relative process number starting at 1.
1988
1989 The rest of the line starting after the first colon follows the "typed output
1990 format" described in the section above. In short, the second column (after the
1991 first ':') indicates the origin, nature and scope of the variable. The third
1992 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1993 "str". Then the fourth column is the value itself, which the consumer knows
1994 how to parse thanks to column 3 and how to process thanks to column 2.
1995
1996 Thus the overall line format in typed mode is :
1997
1998 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
1999
2000 Example :
2001
2002 > show info
2003 Name: HAProxy
2004 Version: 1.7-dev1-de52ea-146
2005 Release_date: 2016/03/11
2006 Nbproc: 1
2007 Process_num: 1
2008 Pid: 28105
2009 Uptime: 0d 0h00m04s
2010 Uptime_sec: 4
2011 Memmax_MB: 0
2012 PoolAlloc_MB: 0
2013 PoolUsed_MB: 0
2014 PoolFailed: 0
2015 (...)
2016
2017 > show info typed
2018 0.Name.1:POS:str:HAProxy
2019 1.Version.1:POS:str:1.7-dev1-de52ea-146
2020 2.Release_date.1:POS:str:2016/03/11
2021 3.Nbproc.1:CGS:u32:1
2022 4.Process_num.1:KGP:u32:1
2023 5.Pid.1:SGP:u32:28105
2024 6.Uptime.1:MDP:str:0d 0h00m08s
2025 7.Uptime_sec.1:MDP:u32:8
2026 8.Memmax_MB.1:CLP:u32:0
2027 9.PoolAlloc_MB.1:MGP:u32:0
2028 10.PoolUsed_MB.1:MGP:u32:0
2029 11.PoolFailed.1:MCP:u32:0
2030 (...)
2031
Simon Horman1084a362016-11-21 17:00:24 +01002032 In the typed format, the presence of the process ID at the end of the
2033 first column makes it very easy to visually aggregate outputs from
2034 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002035 Example :
2036
2037 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
2038 echo show info typed | socat /var/run/haproxy.sock2 ) | \
2039 sort -t . -k 1,1n -k 2,2 -k 3,3n
2040 0.Name.1:POS:str:HAProxy
2041 0.Name.2:POS:str:HAProxy
2042 1.Version.1:POS:str:1.7-dev1-868ab3-148
2043 1.Version.2:POS:str:1.7-dev1-868ab3-148
2044 2.Release_date.1:POS:str:2016/03/11
2045 2.Release_date.2:POS:str:2016/03/11
2046 3.Nbproc.1:CGS:u32:2
2047 3.Nbproc.2:CGS:u32:2
2048 4.Process_num.1:KGP:u32:1
2049 4.Process_num.2:KGP:u32:2
2050 5.Pid.1:SGP:u32:30120
2051 5.Pid.2:SGP:u32:30121
2052 6.Uptime.1:MDP:str:0d 0h01m28s
2053 6.Uptime.2:MDP:str:0d 0h01m28s
2054 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002055
Simon Horman05ee2132017-01-04 09:37:25 +01002056 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002057 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002058
2059 The JSON output contains no extra whitespace in order to reduce the
2060 volume of output. For human consumption passing the output through a
2061 pretty printer may be helpful. Example :
2062
2063 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2064 python -m json.tool
2065
Simon Horman6f6bb382017-01-04 09:37:26 +01002066 The JSON output contains no extra whitespace in order to reduce the
2067 volume of output. For human consumption passing the output through a
2068 pretty printer may be helpful. Example :
2069
2070 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2071 python -m json.tool
2072
Willy Tarreau44aed902015-10-13 14:45:29 +02002073show map [<map>]
2074 Dump info about map converters. Without argument, the list of all available
2075 maps is returned. If a <map> is specified, its contents are dumped. <map> is
2076 the #<id> or <file>. The first column is a unique identifier. It can be used
2077 as reference for the operation "del map" and "set map". The second column is
2078 the pattern and the third column is the sample if available. The data returned
2079 are not directly a list of available maps, but are the list of all patterns
2080 composing any map. Many of these patterns can be shared with ACL.
2081
Willy Tarreau44aed902015-10-13 14:45:29 +02002082show pools
2083 Dump the status of internal memory pools. This is useful to track memory
2084 usage when suspecting a memory leak for example. It does exactly the same
2085 as the SIGQUIT when running in foreground except that it does not flush
2086 the pools.
2087
Willy Tarreau75c62c22018-11-22 11:02:09 +01002088show profiling
2089 Dumps the current profiling settings, one per line, as well as the command
2090 needed to change them.
2091
Willy Tarreau44aed902015-10-13 14:45:29 +02002092show servers state [<backend>]
2093 Dump the state of the servers found in the running configuration. A backend
2094 name or identifier may be provided to limit the output to this backend only.
2095
2096 The dump has the following format:
2097 - first line contains the format version (1 in this specification);
2098 - second line contains the column headers, prefixed by a sharp ('#');
2099 - third line and next ones contain data;
2100 - each line starting by a sharp ('#') is considered as a comment.
2101
Dan Lloyd8e48b872016-07-01 21:01:18 -04002102 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002103 fields and their order per file format version :
2104 1:
2105 be_id: Backend unique id.
2106 be_name: Backend label.
2107 srv_id: Server unique id (in the backend).
2108 srv_name: Server label.
2109 srv_addr: Server IP address.
2110 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002111 0 = SRV_ST_STOPPED
2112 The server is down.
2113 1 = SRV_ST_STARTING
2114 The server is warming up (up but
2115 throttled).
2116 2 = SRV_ST_RUNNING
2117 The server is fully up.
2118 3 = SRV_ST_STOPPING
2119 The server is up but soft-stopping
2120 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002121 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002122 The state is actually a mask of values :
2123 0x01 = SRV_ADMF_FMAINT
2124 The server was explicitly forced into
2125 maintenance.
2126 0x02 = SRV_ADMF_IMAINT
2127 The server has inherited the maintenance
2128 status from a tracked server.
2129 0x04 = SRV_ADMF_CMAINT
2130 The server is in maintenance because of
2131 the configuration.
2132 0x08 = SRV_ADMF_FDRAIN
2133 The server was explicitly forced into
2134 drain state.
2135 0x10 = SRV_ADMF_IDRAIN
2136 The server has inherited the drain status
2137 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002138 0x20 = SRV_ADMF_RMAINT
2139 The server is in maintenance because of an
2140 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002141 0x40 = SRV_ADMF_HMAINT
2142 The server FQDN was set from stats socket.
2143
Willy Tarreau44aed902015-10-13 14:45:29 +02002144 srv_uweight: User visible server's weight.
2145 srv_iweight: Server's initial weight.
2146 srv_time_since_last_change: Time since last operational change.
2147 srv_check_status: Last health check status.
2148 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002149 0 = CHK_RES_UNKNOWN
2150 Initialized to this by default.
2151 1 = CHK_RES_NEUTRAL
2152 Valid check but no status information.
2153 2 = CHK_RES_FAILED
2154 Check failed.
2155 3 = CHK_RES_PASSED
2156 Check succeeded and server is fully up
2157 again.
2158 4 = CHK_RES_CONDPASS
2159 Check reports the server doesn't want new
2160 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002161 srv_check_health: Checks rise / fall current counter.
2162 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002163 The state is actually a mask of values :
2164 0x01 = CHK_ST_INPROGRESS
2165 A check is currently running.
2166 0x02 = CHK_ST_CONFIGURED
2167 This check is configured and may be
2168 enabled.
2169 0x04 = CHK_ST_ENABLED
2170 This check is currently administratively
2171 enabled.
2172 0x08 = CHK_ST_PAUSED
2173 Checks are paused because of maintenance
2174 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002175 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002176 This state uses the same mask values as
2177 "srv_check_state", adding this specific one :
2178 0x10 = CHK_ST_AGENT
2179 Check is an agent check (otherwise it's a
2180 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002181 bk_f_forced_id: Flag to know if the backend ID is forced by
2182 configuration.
2183 srv_f_forced_id: Flag to know if the server's ID is forced by
2184 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002185 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002186 srv_port: Server port.
Baptiste Assmann6d0f38f2018-07-02 17:00:54 +02002187 srvrecord: DNS SRV record associated to this SRV.
Willy Tarreau44aed902015-10-13 14:45:29 +02002188
2189show sess
2190 Dump all known sessions. Avoid doing this on slow connections as this can
2191 be huge. This command is restricted and can only be issued on sockets
2192 configured for levels "operator" or "admin".
2193
2194show sess <id>
2195 Display a lot of internal information about the specified session identifier.
2196 This identifier is the first field at the beginning of the lines in the dumps
2197 of "show sess" (it corresponds to the session pointer). Those information are
2198 useless to most users but may be used by haproxy developers to troubleshoot a
2199 complex bug. The output format is intentionally not documented so that it can
2200 freely evolve depending on demands. You may find a description of all fields
2201 returned in src/dumpstats.c
2202
2203 The special id "all" dumps the states of all sessions, which must be avoided
2204 as much as possible as it is highly CPU intensive and can take a lot of time.
2205
Simon Horman05ee2132017-01-04 09:37:25 +01002206show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
2207 Dump statistics using the CSV format; using the extended typed output
2208 format described in the section above if "typed" is passed after the
2209 other arguments; or in JSON if "json" is passed after the other arguments
2210 . By passing <id>, <type> and <sid>, it is possible to dump only selected
2211 items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002212 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2213 <proxy> may be specified. In this case, this proxy's ID will be used as
2214 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002215 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2216 backends, 4 for servers, -1 for everything. These values can be ORed,
2217 for example:
2218 1 + 2 = 3 -> frontend + backend.
2219 1 + 2 + 4 = 7 -> frontend + backend + server.
2220 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2221
2222 Example :
2223 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2224 >>> Name: HAProxy
2225 Version: 1.4-dev2-49
2226 Release_date: 2009/09/23
2227 Nbproc: 1
2228 Process_num: 1
2229 (...)
2230
2231 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2232 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2233 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2234 (...)
2235 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2236
2237 $
2238
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002239 In this example, two commands have been issued at once. That way it's easy to
2240 find which process the stats apply to in multi-process mode. This is not
2241 needed in the typed output format as the process number is reported on each
2242 line. Notice the empty line after the information output which marks the end
2243 of the first block. A similar empty line appears at the end of the second
2244 block (stats) so that the reader knows the output has not been truncated.
2245
2246 When "typed" is specified, the output format is more suitable to monitoring
2247 tools because it provides numeric positions and indicates the type of each
2248 output field. Each value stands on its own line with process number, element
2249 number, nature, origin and scope. This same format is available via the HTTP
2250 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002251 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002252 is no need for a consumer to store everything at once.
2253
2254 When using the typed output format, each line is made of 4 columns delimited
2255 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2256 first element is a letter indicating the type of the object being described.
2257 At the moment the following object types are known : 'F' for a frontend, 'B'
2258 for a backend, 'L' for a listener, and 'S' for a server. The second element
2259 The second element is a positive integer representing the unique identifier of
2260 the proxy the object belongs to. It is equivalent to the "iid" column of the
2261 CSV output and matches the value in front of the optional "id" directive found
2262 in the frontend or backend section. The third element is a positive integer
2263 containing the unique object identifier inside the proxy, and corresponds to
2264 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2265 or a backend. For a listener or a server, this corresponds to their respective
2266 ID inside the proxy. The fourth element is the numeric position of the field
2267 in the list (starting at zero). This position shall not change over time, but
2268 holes are to be expected, depending on build options or if some fields are
2269 deleted in the future. The fifth element is the field name as it appears in
2270 the CSV output. The sixth element is a positive integer and is the relative
2271 process number starting at 1.
2272
2273 The rest of the line starting after the first colon follows the "typed output
2274 format" described in the section above. In short, the second column (after the
2275 first ':') indicates the origin, nature and scope of the variable. The third
2276 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2277 "str". Then the fourth column is the value itself, which the consumer knows
2278 how to parse thanks to column 3 and how to process thanks to column 2.
2279
2280 Thus the overall line format in typed mode is :
2281
2282 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2283
2284 Here's an example of typed output format :
2285
2286 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2287 F.2.0.0.pxname.1:MGP:str:private-frontend
2288 F.2.0.1.svname.1:MGP:str:FRONTEND
2289 F.2.0.8.bin.1:MGP:u64:0
2290 F.2.0.9.bout.1:MGP:u64:0
2291 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2292 L.2.1.0.pxname.1:MGP:str:private-frontend
2293 L.2.1.1.svname.1:MGP:str:sock-1
2294 L.2.1.17.status.1:MGP:str:OPEN
2295 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2296 S.3.13.60.rtime.1:MCP:u32:0
2297 S.3.13.61.ttime.1:MCP:u32:0
2298 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2299 S.3.13.64.agent_duration.1:MGP:u64:2001
2300 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2301 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2302 S.3.13.67.check_rise.1:MCP:u32:2
2303 S.3.13.68.check_fall.1:MCP:u32:3
2304 S.3.13.69.check_health.1:SGP:u32:0
2305 S.3.13.70.agent_rise.1:MaP:u32:1
2306 S.3.13.71.agent_fall.1:SGP:u32:1
2307 S.3.13.72.agent_health.1:SGP:u32:1
2308 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2309 S.3.13.75.mode.1:MAP:str:http
2310 B.3.0.0.pxname.1:MGP:str:private-backend
2311 B.3.0.1.svname.1:MGP:str:BACKEND
2312 B.3.0.2.qcur.1:MGP:u32:0
2313 B.3.0.3.qmax.1:MGP:u32:0
2314 B.3.0.4.scur.1:MGP:u32:0
2315 B.3.0.5.smax.1:MGP:u32:0
2316 B.3.0.6.slim.1:MGP:u32:1000
2317 B.3.0.55.lastsess.1:MMP:s32:-1
2318 (...)
2319
Simon Horman1084a362016-11-21 17:00:24 +01002320 In the typed format, the presence of the process ID at the end of the
2321 first column makes it very easy to visually aggregate outputs from
2322 multiple processes, as show in the example below where each line appears
2323 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002324
2325 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2326 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2327 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2328 B.3.0.0.pxname.1:MGP:str:private-backend
2329 B.3.0.0.pxname.2:MGP:str:private-backend
2330 B.3.0.1.svname.1:MGP:str:BACKEND
2331 B.3.0.1.svname.2:MGP:str:BACKEND
2332 B.3.0.2.qcur.1:MGP:u32:0
2333 B.3.0.2.qcur.2:MGP:u32:0
2334 B.3.0.3.qmax.1:MGP:u32:0
2335 B.3.0.3.qmax.2:MGP:u32:0
2336 B.3.0.4.scur.1:MGP:u32:0
2337 B.3.0.4.scur.2:MGP:u32:0
2338 B.3.0.5.smax.1:MGP:u32:0
2339 B.3.0.5.smax.2:MGP:u32:0
2340 B.3.0.6.slim.1:MGP:u32:1000
2341 B.3.0.6.slim.2:MGP:u32:1000
2342 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002343
Simon Horman05ee2132017-01-04 09:37:25 +01002344 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002345 using "show schema json".
2346
2347 The JSON output contains no extra whitespace in order to reduce the
2348 volume of output. For human consumption passing the output through a
2349 pretty printer may be helpful. Example :
2350
2351 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2352 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002353
2354 The JSON output contains no extra whitespace in order to reduce the
2355 volume of output. For human consumption passing the output through a
2356 pretty printer may be helpful. Example :
2357
2358 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2359 python -m json.tool
2360
Willy Tarreau44aed902015-10-13 14:45:29 +02002361show stat resolvers [<resolvers section id>]
2362 Dump statistics for the given resolvers section, or all resolvers sections
2363 if no section is supplied.
2364
2365 For each name server, the following counters are reported:
2366 sent: number of DNS requests sent to this server
2367 valid: number of DNS valid responses received from this server
2368 update: number of DNS responses used to update the server's IP address
2369 cname: number of CNAME responses
2370 cname_error: CNAME errors encountered with this server
2371 any_err: number of empty response (IE: server does not support ANY type)
2372 nx: non existent domain response received from this server
2373 timeout: how many time this server did not answer in time
2374 refused: number of requests refused by this server
2375 other: any other DNS errors
2376 invalid: invalid DNS response (from a protocol point of view)
2377 too_big: too big response
2378 outdated: number of response arrived too late (after an other name server)
2379
2380show table
2381 Dump general information on all known stick-tables. Their name is returned
2382 (the name of the proxy which holds them), their type (currently zero, always
2383 IP), their size in maximum possible number of entries, and the number of
2384 entries currently in use.
2385
2386 Example :
2387 $ echo "show table" | socat stdio /tmp/sock1
2388 >>> # table: front_pub, type: ip, size:204800, used:171454
2389 >>> # table: back_rdp, type: ip, size:204800, used:0
2390
2391show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2392 Dump contents of stick-table <name>. In this mode, a first line of generic
2393 information about the table is reported as with "show table", then all
2394 entries are dumped. Since this can be quite heavy, it is possible to specify
2395 a filter in order to specify what entries to display.
2396
2397 When the "data." form is used the filter applies to the stored data (see
2398 "stick-table" in section 4.2). A stored data type must be specified
2399 in <type>, and this data type must be stored in the table otherwise an
2400 error is reported. The data is compared according to <operator> with the
2401 64-bit integer <value>. Operators are the same as with the ACLs :
2402
2403 - eq : match entries whose data is equal to this value
2404 - ne : match entries whose data is not equal to this value
2405 - le : match entries whose data is less than or equal to this value
2406 - ge : match entries whose data is greater than or equal to this value
2407 - lt : match entries whose data is less than this value
2408 - gt : match entries whose data is greater than this value
2409
2410
2411 When the key form is used the entry <key> is shown. The key must be of the
2412 same type as the table, which currently is limited to IPv4, IPv6, integer,
2413 and string.
2414
2415 Example :
2416 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2417 >>> # table: http_proxy, type: ip, size:204800, used:2
2418 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2419 bytes_out_rate(60000)=187
2420 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2421 bytes_out_rate(60000)=191
2422
2423 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2424 >>> # table: http_proxy, type: ip, size:204800, used:2
2425 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2426 bytes_out_rate(60000)=191
2427
2428 $ echo "show table http_proxy data.conn_rate gt 5" | \
2429 socat stdio /tmp/sock1
2430 >>> # table: http_proxy, type: ip, size:204800, used:2
2431 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2432 bytes_out_rate(60000)=191
2433
2434 $ echo "show table http_proxy key 127.0.0.2" | \
2435 socat stdio /tmp/sock1
2436 >>> # table: http_proxy, type: ip, size:204800, used:2
2437 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2438 bytes_out_rate(60000)=191
2439
2440 When the data criterion applies to a dynamic value dependent on time such as
2441 a bytes rate, the value is dynamically computed during the evaluation of the
2442 entry in order to decide whether it has to be dumped or not. This means that
2443 such a filter could match for some time then not match anymore because as
2444 time goes, the average event rate drops.
2445
2446 It is possible to use this to extract lists of IP addresses abusing the
2447 service, in order to monitor them or even blacklist them in a firewall.
2448 Example :
2449 $ echo "show table http_proxy data.gpc0 gt 0" \
2450 | socat stdio /tmp/sock1 \
2451 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2452 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2453
William Lallemandbb933462016-05-31 21:09:53 +02002454show tls-keys [id|*]
2455 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2456 and the file from which the keys have been loaded is shown. Both of those
2457 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2458 specified as parameter, it will dump the tickets, using * it will dump every
2459 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002460
Simon Horman6f6bb382017-01-04 09:37:26 +01002461show schema json
2462 Dump the schema used for the output of "show info json" and "show stat json".
2463
2464 The contains no extra whitespace in order to reduce the volume of output.
2465 For human consumption passing the output through a pretty printer may be
2466 helpful. Example :
2467
2468 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
2469 python -m json.tool
2470
2471 The schema follows "JSON Schema" (json-schema.org) and accordingly
2472 verifiers may be used to verify the output of "show info json" and "show
2473 stat json" against the schema.
2474
2475
Willy Tarreau44aed902015-10-13 14:45:29 +02002476shutdown frontend <frontend>
2477 Completely delete the specified frontend. All the ports it was bound to will
2478 be released. It will not be possible to enable the frontend anymore after
2479 this operation. This is intended to be used in environments where stopping a
2480 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2481 way it's possible to release the port and bind it into another process to
2482 restore operations. The frontend will not appear at all on the stats page
2483 once it is terminated.
2484
2485 The frontend may be specified either by its name or by its numeric ID,
2486 prefixed with a sharp ('#').
2487
2488 This command is restricted and can only be issued on sockets configured for
2489 level "admin".
2490
2491shutdown session <id>
2492 Immediately terminate the session matching the specified session identifier.
2493 This identifier is the first field at the beginning of the lines in the dumps
2494 of "show sess" (it corresponds to the session pointer). This can be used to
2495 terminate a long-running session without waiting for a timeout or when an
2496 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2497 flag in the logs.
2498
2499shutdown sessions server <backend>/<server>
2500 Immediately terminate all the sessions attached to the specified server. This
2501 can be used to terminate long-running sessions after a server is put into
2502 maintenance mode, for instance. Such terminated sessions are reported with a
2503 'K' flag in the logs.
2504
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002505
William Lallemand142db372018-12-11 18:56:45 +010025069.4. Master CLI
2507---------------
2508
2509The master CLI is a socket bound to the master process in master-worker mode.
2510This CLI gives access to the unix socket commands in every running or leaving
2511processes and allows a basic supervision of those processes.
2512
2513The master CLI is configurable only from the haproxy program arguments with
2514the -S option. This option also takes bind options separated by commas.
2515
2516Example:
2517
2518 # haproxy -W -S 127.0.0.1:1234 -f test1.cfg
2519 # haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
William Lallemandb7ea1412018-12-13 09:05:47 +01002520 # haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
William Lallemand142db372018-12-11 18:56:45 +01002521
2522The master CLI introduces a new 'show proc' command to surpervise the
2523processes:
2524
2525Example:
2526
2527 $ echo 'show proc' | socat /var/run/haproxy-master.sock -
2528 #<PID> <type> <relative PID> <reloads> <uptime>
2529 1162 master 0 5 0d 00h02m07s
2530 # workers
2531 1271 worker 1 0 0d 00h00m00s
2532 1272 worker 2 0 0d 00h00m00s
2533 # old workers
William Lallemand256bf0d2018-12-12 13:45:57 +01002534 1233 worker [was: 1] 3 0d 00h00m43s
William Lallemand142db372018-12-11 18:56:45 +01002535
2536
2537In this example, the master has been reloaded 5 times but one of the old
2538worker is still running and survived 3 reloads. You could access the CLI of
2539this worker to understand what's going on.
2540
Willy Tarreau52880f92018-12-15 13:30:03 +01002541When the prompt is enabled (via the "prompt" command), the context the CLI is
2542working on is displayed in the prompt. The master is identified by the "master"
2543string, and other processes are identified with their PID. In case the last
2544reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
2545that it becomes visible that the process is still running on the previous
2546configuration and that the new configuration is not operational.
2547
William Lallemand142db372018-12-11 18:56:45 +01002548The master CLI uses a special prefix notation to access the multiple
2549processes. This notation is easily identifiable as it begins by a @.
2550
2551A @ prefix can be followed by a relative process number or by an exclamation
2552point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
2553master. Leaving processes are only accessible with the PID as relative process
2554number are only usable with the current processes.
2555
2556Examples:
2557
2558 $ socat /var/run/haproxy-master.sock readline
2559 prompt
2560 master> @1 show info; @2 show info
2561 [...]
2562 Process_num: 1
2563 Pid: 1271
2564 [...]
2565 Process_num: 2
2566 Pid: 1272
2567 [...]
2568 master>
2569
2570 $ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
2571 [...]
2572
2573A prefix could be use as a command, which will send every next commands to
2574the specified process.
2575
2576Examples:
2577
2578 $ socat /var/run/haproxy-master.sock readline
2579 prompt
2580 master> @1
2581 1271> show info
2582 [...]
2583 1271> show stat
2584 [...]
2585 1271> @
2586 master>
2587
2588 $ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
2589 [...]
2590
William Lallemanda57b7e32018-12-14 21:11:31 +01002591You can also reload the HAProxy master process with the "reload" command which
2592does the same as a `kill -USR2` on the master process, provided that the user
2593has at least "operator" or "admin" privileges.
2594
2595Example:
2596
2597 $ echo "reload" | socat /var/run/haproxy-master.sock
2598
2599Note that a reload will close the connection to the master CLI.
2600
William Lallemand142db372018-12-11 18:56:45 +01002601
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200260210. Tricks for easier configuration management
2603----------------------------------------------
2604
2605It is very common that two HAProxy nodes constituting a cluster share exactly
2606the same configuration modulo a few addresses. Instead of having to maintain a
2607duplicate configuration for each node, which will inevitably diverge, it is
2608possible to include environment variables in the configuration. Thus multiple
2609configuration may share the exact same file with only a few different system
2610wide environment variables. This started in version 1.5 where only addresses
2611were allowed to include environment variables, and 1.6 goes further by
2612supporting environment variables everywhere. The syntax is the same as in the
2613UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2614curly brace ('{'), then the variable name followed by the closing brace ('}').
2615Except for addresses, environment variables are only interpreted in arguments
2616surrounded with double quotes (this was necessary not to break existing setups
2617using regular expressions involving the dollar symbol).
2618
2619Environment variables also make it convenient to write configurations which are
2620expected to work on various sites where only the address changes. It can also
2621permit to remove passwords from some configs. Example below where the the file
2622"site1.env" file is sourced by the init script upon startup :
2623
2624 $ cat site1.env
2625 LISTEN=192.168.1.1
2626 CACHE_PFX=192.168.11
2627 SERVER_PFX=192.168.22
2628 LOGGER=192.168.33.1
2629 STATSLP=admin:pa$$w0rd
2630 ABUSERS=/etc/haproxy/abuse.lst
2631 TIMEOUT=10s
2632
2633 $ cat haproxy.cfg
2634 global
2635 log "${LOGGER}:514" local0
2636
2637 defaults
2638 mode http
2639 timeout client "${TIMEOUT}"
2640 timeout server "${TIMEOUT}"
2641 timeout connect 5s
2642
2643 frontend public
2644 bind "${LISTEN}:80"
2645 http-request reject if { src -f "${ABUSERS}" }
2646 stats uri /stats
2647 stats auth "${STATSLP}"
2648 use_backend cache if { path_end .jpg .css .ico }
2649 default_backend server
2650
2651 backend cache
2652 server cache1 "${CACHE_PFX}.1:18080" check
2653 server cache2 "${CACHE_PFX}.2:18080" check
2654
2655 backend server
2656 server cache1 "${SERVER_PFX}.1:8080" check
2657 server cache2 "${SERVER_PFX}.2:8080" check
2658
2659
266011. Well-known traps to avoid
2661-----------------------------
2662
2663Once in a while, someone reports that after a system reboot, the haproxy
2664service wasn't started, and that once they start it by hand it works. Most
2665often, these people are running a clustered IP address mechanism such as
2666keepalived, to assign the service IP address to the master node only, and while
2667it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2668working after they bound it to the virtual IP address. What happens here is
2669that when the service starts, the virtual IP address is not yet owned by the
2670local node, so when HAProxy wants to bind to it, the system rejects this
2671because it is not a local IP address. The fix doesn't consist in delaying the
2672haproxy service startup (since it wouldn't stand a restart), but instead to
2673properly configure the system to allow binding to non-local addresses. This is
2674easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2675is also needed in order to transparently intercept the IP traffic that passes
2676through HAProxy for a specific target address.
2677
2678Multi-process configurations involving source port ranges may apparently seem
2679to work but they will cause some random failures under high loads because more
2680than one process may try to use the same source port to connect to the same
2681server, which is not possible. The system will report an error and a retry will
2682happen, picking another port. A high value in the "retries" parameter may hide
2683the effect to a certain extent but this also comes with increased CPU usage and
2684processing time. Logs will also report a certain number of retries. For this
2685reason, port ranges should be avoided in multi-process configurations.
2686
Dan Lloyd8e48b872016-07-01 21:01:18 -04002687Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002688processes bound to the same IP:port, during troubleshooting it can happen that
2689an old process was not stopped before a new one was started. This provides
2690absurd test results which tend to indicate that any change to the configuration
2691is ignored. The reason is that in fact even the new process is restarted with a
2692new configuration, the old one also gets some incoming connections and
2693processes them, returning unexpected results. When in doubt, just stop the new
2694process and try again. If it still works, it very likely means that an old
2695process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2696help here.
2697
2698When adding entries to an ACL from the command line (eg: when blacklisting a
2699source address), it is important to keep in mind that these entries are not
2700synchronized to the file and that if someone reloads the configuration, these
2701updates will be lost. While this is often the desired effect (for blacklisting)
2702it may not necessarily match expectations when the change was made as a fix for
2703a problem. See the "add acl" action of the CLI interface.
2704
2705
270612. Debugging and performance issues
2707------------------------------------
2708
2709When HAProxy is started with the "-d" option, it will stay in the foreground
2710and will print one line per event, such as an incoming connection, the end of a
2711connection, and for each request or response header line seen. This debug
2712output is emitted before the contents are processed, so they don't consider the
2713local modifications. The main use is to show the request and response without
2714having to run a network sniffer. The output is less readable when multiple
2715connections are handled in parallel, though the "debug2ansi" and "debug2html"
2716scripts found in the examples/ directory definitely help here by coloring the
2717output.
2718
2719If a request or response is rejected because HAProxy finds it is malformed, the
2720best thing to do is to connect to the CLI and issue "show errors", which will
2721report the last captured faulty request and response for each frontend and
2722backend, with all the necessary information to indicate precisely the first
2723character of the input stream that was rejected. This is sometimes needed to
2724prove to customers or to developers that a bug is present in their code. In
2725this case it is often possible to relax the checks (but still keep the
2726captures) using "option accept-invalid-http-request" or its equivalent for
2727responses coming from the server "option accept-invalid-http-response". Please
2728see the configuration manual for more details.
2729
2730Example :
2731
2732 > show errors
2733 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2734
2735 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2736 backend <NONE> (#-1), server <NONE> (#-1), event #0
2737 src 127.0.0.1:51981, session #0, session flags 0x00000080
2738 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2739 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2740 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2741 pending 31 bytes, wrapping at 8040, error at position 13:
2742
2743 00000 GET /invalid request HTTP/1.1\r\n
2744
2745
2746The output of "show info" on the CLI provides a number of useful information
2747regarding the maximum connection rate ever reached, maximum SSL key rate ever
2748reached, and in general all information which can help to explain temporary
2749issues regarding CPU or memory usage. Example :
2750
2751 > show info
2752 Name: HAProxy
2753 Version: 1.6-dev7-e32d18-17
2754 Release_date: 2015/10/12
2755 Nbproc: 1
2756 Process_num: 1
2757 Pid: 7949
2758 Uptime: 0d 0h02m39s
2759 Uptime_sec: 159
2760 Memmax_MB: 0
2761 Ulimit-n: 120032
2762 Maxsock: 120032
2763 Maxconn: 60000
2764 Hard_maxconn: 60000
2765 CurrConns: 0
2766 CumConns: 3
2767 CumReq: 3
2768 MaxSslConns: 0
2769 CurrSslConns: 0
2770 CumSslConns: 0
2771 Maxpipes: 0
2772 PipesUsed: 0
2773 PipesFree: 0
2774 ConnRate: 0
2775 ConnRateLimit: 0
2776 MaxConnRate: 1
2777 SessRate: 0
2778 SessRateLimit: 0
2779 MaxSessRate: 1
2780 SslRate: 0
2781 SslRateLimit: 0
2782 MaxSslRate: 0
2783 SslFrontendKeyRate: 0
2784 SslFrontendMaxKeyRate: 0
2785 SslFrontendSessionReuse_pct: 0
2786 SslBackendKeyRate: 0
2787 SslBackendMaxKeyRate: 0
2788 SslCacheLookups: 0
2789 SslCacheMisses: 0
2790 CompressBpsIn: 0
2791 CompressBpsOut: 0
2792 CompressBpsRateLim: 0
2793 ZlibMemUsage: 0
2794 MaxZlibMemUsage: 0
2795 Tasks: 5
2796 Run_queue: 1
2797 Idle_pct: 100
2798 node: wtap
2799 description:
2800
2801When an issue seems to randomly appear on a new version of HAProxy (eg: every
2802second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04002803memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002804filling of the memory area with a configurable byte. By default this byte is
28050x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2806will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04002807Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002808slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04002809an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002810byte zero, it clearly means you've found a bug and you definitely need to
2811report it. Otherwise if there's no clear change, the problem it is not related.
2812
2813When debugging some latency issues, it is important to use both strace and
2814tcpdump on the local machine, and another tcpdump on the remote system. The
2815reason for this is that there are delays everywhere in the processing chain and
2816it is important to know which one is causing latency to know where to act. In
2817practice, the local tcpdump will indicate when the input data come in. Strace
2818will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2819openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2820show when haproxy sends the data, and tcpdump will show when the system sends
2821these data to the interface. Then the external tcpdump will show when the data
2822sent are really received (since the local one only shows when the packets are
2823queued). The benefit of sniffing on the local system is that strace and tcpdump
2824will use the same reference clock. Strace should be used with "-tts200" to get
2825complete timestamps and report large enough chunks of data to read them.
2826Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2827numbers and complete timestamps.
2828
2829In practice, received data are almost always immediately received by haproxy
2830(unless the machine has a saturated CPU or these data are invalid and not
2831delivered). If these data are received but not sent, it generally is because
2832the output buffer is saturated (ie: recipient doesn't consume the data fast
2833enough). This can be confirmed by seeing that the polling doesn't notify of
2834the ability to write on the output file descriptor for some time (it's often
2835easier to spot in the strace output when the data finally leave and then roll
2836back to see when the write event was notified). It generally matches an ACK
2837received from the recipient, and detected by tcpdump. Once the data are sent,
2838they may spend some time in the system doing nothing. Here again, the TCP
2839congestion window may be limited and not allow these data to leave, waiting for
2840an ACK to open the window. If the traffic is idle and the data take 40 ms or
2841200 ms to leave, it's a different issue (which is not an issue), it's the fact
2842that the Nagle algorithm prevents empty packets from leaving immediately, in
2843hope that they will be merged with subsequent data. HAProxy automatically
2844disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2845enabled when forwarding an HTTP body (and this contributes to the performance
2846improvement there by reducing the number of packets). Some HTTP non-compliant
2847applications may be sensitive to the latency when delivering incomplete HTTP
2848response messages. In this case you will have to enable "option http-no-delay"
2849to disable Nagle in order to work around their design, keeping in mind that any
2850other proxy in the chain may similarly be impacted. If tcpdump reports that data
2851leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04002852is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002853preventing the data from leaving, or more commonly that HAProxy is in fact
2854running in a virtual machine and that for whatever reason the hypervisor has
2855decided that the data didn't need to be sent immediately. In virtualized
2856environments, latency issues are almost always caused by the virtualization
2857layer, so in order to save time, it's worth first comparing tcpdump in the VM
2858and on the external components. Any difference has to be credited to the
2859hypervisor and its accompanying drivers.
2860
2861When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2862means that the side sending them has got the proof of a lost packet. While not
2863seeing them doesn't mean there are no losses, seeing them definitely means the
2864network is lossy. Losses are normal on a network, but at a rate where SACKs are
2865not noticeable at the naked eye. If they appear a lot in the traces, it is
2866worth investigating exactly what happens and where the packets are lost. HTTP
2867doesn't cope well with TCP losses, which introduce huge latencies.
2868
2869The "netstat -i" command will report statistics per interface. An interface
2870where the Rx-Ovr counter grows indicates that the system doesn't have enough
2871resources to receive all incoming packets and that they're lost before being
2872processed by the network driver. Rx-Drp indicates that some received packets
2873were lost in the network stack because the application doesn't process them
2874fast enough. This can happen during some attacks as well. Tx-Drp means that
2875the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04002876should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002877
2878
287913. Security considerations
2880---------------------------
2881
2882HAProxy is designed to run with very limited privileges. The standard way to
2883use it is to isolate it into a chroot jail and to drop its privileges to a
2884non-root user without any permissions inside this jail so that if any future
2885vulnerability were to be discovered, its compromise would not affect the rest
2886of the system.
2887
Dan Lloyd8e48b872016-07-01 21:01:18 -04002888In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002889pointless to build hand-made chroots to start the process there, these ones are
2890painful to build, are never properly maintained and always contain way more
2891bugs than the main file-system. And in case of compromise, the intruder can use
2892the purposely built file-system. Unfortunately many administrators confuse
2893"start as root" and "run as root", resulting in the uid change to be done prior
2894to starting haproxy, and reducing the effective security restrictions.
2895
2896HAProxy will need to be started as root in order to :
2897 - adjust the file descriptor limits
2898 - bind to privileged port numbers
2899 - bind to a specific network interface
2900 - transparently listen to a foreign address
2901 - isolate itself inside the chroot jail
2902 - drop to another non-privileged UID
2903
2904HAProxy may require to be run as root in order to :
2905 - bind to an interface for outgoing connections
2906 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04002907 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002908
2909Most users will never need the "run as root" case. But the "start as root"
2910covers most usages.
2911
2912A safe configuration will have :
2913
2914 - a chroot statement pointing to an empty location without any access
2915 permissions. This can be prepared this way on the UNIX command line :
2916
2917 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
2918
2919 and referenced like this in the HAProxy configuration's global section :
2920
2921 chroot /var/empty
2922
2923 - both a uid/user and gid/group statements in the global section :
2924
2925 user haproxy
2926 group haproxy
2927
2928 - a stats socket whose mode, uid and gid are set to match the user and/or
2929 group allowed to access the CLI so that nobody may access it :
2930
2931 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
2932