blob: 119c3fa6da71ff180dae58228372cca549d8af18 [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau8234f6d2016-03-14 00:10:05 +01004 version 1.7
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003310. Tricks for easier configuration management
3411. Well-known traps to avoid
3512. Debugging and performance issues
3613. Security considerations
37
38
391. Prerequisites
40----------------
41
42In this document it is assumed that the reader has sufficient administration
43skills on a UNIX-like operating system, uses the shell on a daily basis and is
44familiar with troubleshooting utilities such as strace and tcpdump.
45
46
472. Quick reminder about HAProxy's architecture
48----------------------------------------------
49
50HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is
51uses event multiplexing to schedule all of its activities instead of relying on
52the system to schedule between multiple activities. Most of the time it runs as
53a single process, so the output of "ps aux" on a system will report only one
54"haproxy" process, unless a soft reload is in progress and an older process is
55finishing its job in parallel to the new one. It is thus always easy to trace
56its activity using the strace utility.
57
58HAProxy is designed to isolate itself into a chroot jail during startup, where
59it cannot perform any file-system access at all. This is also true for the
60libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
61a running process will not be able to reload a configuration file to apply
62changes, instead a new process will be started using the updated configuration
63file. Some other less obvious effects are that some timezone files or resolver
64files the libc might attempt to access at run time will not be found, though
65this should generally not happen as they're not needed after startup. A nice
66consequence of this principle is that the HAProxy process is totally stateless,
67and no cleanup is needed after it's killed, so any killing method that works
68will do the right thing.
69
70HAProxy doesn't write log files, but it relies on the standard syslog protocol
71to send logs to a remote server (which is often located on the same system).
72
73HAProxy uses its internal clock to enforce timeouts, that is derived from the
74system's time but where unexpected drift is corrected. This is done by limiting
75the time spent waiting in poll() for an event, and measuring the time it really
76took. In practice it never waits more than one second. This explains why, when
77running strace over a completely idle process, periodic calls to poll() (or any
78of its variants) surrounded by two gettimeofday() calls are noticed. They are
79normal, completely harmless and so cheap that the load they imply is totally
80undetectable at the system scale, so there's nothing abnormal there. Example :
81
82 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
83 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
84 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
85 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
86 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
87 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
88
89HAProxy is a TCP proxy, not a router. It deals with established connections that
90have been validated by the kernel, and not with packets of any form nor with
91sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
92may prevent it from binding a port. It relies on the system to accept incoming
93connections and to initiate outgoing connections. An immediate effect of this is
94that there is no relation between packets observed on the two sides of a
95forwarded connection, which can be of different size, numbers and even family.
96Since a connection may only be accepted from a socket in LISTEN state, all the
97sockets it is listening to are necessarily visible using the "netstat" utility
98to show listening sockets. Example :
99
100 # netstat -ltnp
101 Active Internet connections (only servers)
102 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
103 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
104 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
105 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
106
107
1083. Starting HAProxy
109-------------------
110
111HAProxy is started by invoking the "haproxy" program with a number of arguments
112passed on the command line. The actual syntax is :
113
114 $ haproxy [<options>]*
115
116where [<options>]* is any number of options. An option always starts with '-'
117followed by one of more letters, and possibly followed by one or multiple extra
118arguments. Without any option, HAProxy displays the help page with a reminder
119about supported options. Available options may vary slightly based on the
120operating system. A fair number of these options overlap with an equivalent one
121if the "global" section. In this case, the command line always has precedence
122over the configuration file, so that the command line can be used to quickly
123enforce some settings without touching the configuration files. The current
124list of options is :
125
126 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200127 file/directory to be loaded and processed in the declaration order. It is
128 mostly useful when relying on the shell to load many files that are
129 numerically ordered. See also "-f". The difference between "--" and "-f" is
130 that one "-f" must be placed before each file name, while a single "--" is
131 needed before all file names. Both options can be used together, the
132 command line ordering still applies. When more than one file is specified,
133 each file must start on a section boundary, so the first keyword of each
134 file must be one of "global", "defaults", "peers", "listen", "frontend",
135 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200136
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200137 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
138 loaded. If <cfgdir> is a directory, all the files (and only files) it
Maxime de Roucye3841392016-05-18 23:13:38 +0200139 containes are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200140 configuration files to be loaded ; only files with ".cfg" extension are
141 added, only non hidden files (not prefixed with ".") are added.
142 Configuration files are loaded and processed in their declaration order.
143 This option may be specified multiple times to load multiple files. See
144 also "--". The difference between "--" and "-f" is that one "-f" must be
145 placed before each file name, while a single "--" is needed before all file
146 names. Both options can be used together, the command line ordering still
147 applies. When more than one file is specified, each file must start on a
148 section boundary, so the first keyword of each file must be one of
149 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
150 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200151
152 -C <dir> : changes to directory <dir> before loading configuration
153 files. This is useful when using relative paths. Warning when using
154 wildcards after "--" which are in fact replaced by the shell before
155 starting haproxy.
156
157 -D : start as a daemon. The process detaches from the current terminal after
158 forking, and errors are not reported anymore in the terminal. It is
159 equivalent to the "daemon" keyword in the "global" section of the
160 configuration. It is recommended to always force it in any init script so
161 that a faulty configuration doesn't prevent the system from booting.
162
163 -Ds : work in systemd mode. Only used by the systemd wrapper.
164
165 -L <name> : change the local peer name to <name>, which defaults to the local
166 hostname. This is used only with peers replication.
167
168 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
169 builtin default value (usually 2000). Only useful for debugging.
170
171 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
172 "quiet".
173
174 -c : only performs a check of the configuration files and exits before trying
175 to bind. The exit status is zero if everything is OK, or non-zero if an
176 error is encountered.
177
178 -d : enable debug mode. This disables daemon mode, forces the process to stay
179 in foreground and to show incoming and outgoing events. It is equivalent to
180 the "global" section's "debug" keyword. It must never be used in an init
181 script.
182
183 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
184 can be used when suspecting that getaddrinfo() doesn't work as expected.
185 This option was made available because many bogus implementations of
186 getaddrinfo() exist on various systems and cause anomalies that are
187 difficult to troubleshoot.
188
189 -dM[<byte>] : forces memory poisonning, which means that each and every
190 memory region allocated with malloc() or pool_alloc2() will be filled with
191 <byte> before being passed to the caller. When <byte> is not specified, it
192 defaults to 0x50 ('P'). While this slightly slows down operations, it is
193 useful to reliably trigger issues resulting from missing initializations in
194 the code that cause random crashes. Note that -dM0 has the effect of
195 turning any malloc() into a calloc(). In any case if a bug appears or
196 disappears when using this option it means there is a bug in haproxy, so
197 please report it.
198
199 -dS : disable use of the splice() system call. It is equivalent to the
200 "global" section's "nosplice" keyword. This may be used when splice() is
201 suspected to behave improperly or to cause performance issues, or when
202 using strace to see the forwarded data (which do not appear when using
203 splice()).
204
205 -dV : disable SSL verify on the server side. It is equivalent to having
206 "ssl-server-verify none" in the "global" section. This is useful when
207 trying to reproduce production issues out of the production
208 environment. Never use this in an init script as it degrades SSL security
209 to the servers.
210
211 -db : disable background mode and multi-process mode. The process remains in
212 foreground. It is mainly used during development or during small tests, as
213 Ctrl-C is enough to stop the process. Never use it in an init script.
214
215 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
216 section's keyword "noepoll". It is mostly useful when suspecting a bug
217 related to this poller. On systems supporting epoll, the fallback will
218 generally be the "poll" poller.
219
220 -dk : disable the use of the "kqueue" poller. It is equivalent to the
221 "global" section's keyword "nokqueue". It is mostly useful when suspecting
222 a bug related to this poller. On systems supporting kqueue, the fallback
223 will generally be the "poll" poller.
224
225 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
226 section's keyword "nopoll". It is mostly useful when suspecting a bug
227 related to this poller. On systems supporting poll, the fallback will
228 generally be the "select" poller, which cannot be disabled and is limited
229 to 1024 file descriptors.
230
Willy Tarreau70060452015-12-14 12:46:07 +0100231 -m <limit> : limit the total allocatable memory to <limit> megabytes across
232 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200233 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100234 mostly used to force the processes to work in a constrained resource usage
235 scenario. It is important to note that the memory is not shared between
236 processes, so in a multi-process scenario, this value is first divided by
237 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200238
239 -n <limit> : limits the per-process connection limit to <limit>. This is
240 equivalent to the global section's keyword "maxconn". It has precedence
241 over this keyword. This may be used to quickly force lower limits to avoid
242 a service outage on systems where resource limits are too low.
243
244 -p <file> : write all processes' pids into <file> during startup. This is
245 equivalent to the "global" section's keyword "pidfile". The file is opened
246 before entering the chroot jail, and after doing the chdir() implied by
247 "-C". Each pid appears on its own line.
248
249 -q : set "quiet" mode. This disables some messages during the configuration
250 parsing and during startup. It can be used in combination with "-c" to
251 just check if a configuration file is valid or not.
252
253 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
254 completion to ask them to finish what they are doing and to leave. <pid>
255 is a list of pids to signal (one per argument). The list ends on any
256 option starting with a "-". It is not a problem if the list of pids is
257 empty, so that it can be built on the fly based on the result of a command
258 like "pidof" or "pgrep".
259
260 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
261 boot completion to terminate them immediately without finishing what they
262 were doing. <pid> is a list of pids to signal (one per argument). The list
263 is ends on any option starting with a "-". It is not a problem if the list
264 of pids is empty, so that it can be built on the fly based on the result of
265 a command like "pidof" or "pgrep".
266
267 -v : report the version and build date.
268
269 -vv : display the version, build options, libraries versions and usable
270 pollers. This output is systematically requested when filing a bug report.
271
272A safe way to start HAProxy from an init file consists in forcing the deamon
273mode, storing existing pids to a pid file and using this pid file to notify
274older processes to finish before leaving :
275
276 haproxy -f /etc/haproxy.cfg \
277 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
278
279When the configuration is split into a few specific files (eg: tcp vs http),
280it is recommended to use the "-f" option :
281
282 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
283 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
284 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
285 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
286
287When an unknown number of files is expected, such as customer-specific files,
288it is recommended to assign them a name starting with a fixed-size sequence
289number and to use "--" to load them, possibly after loading some defaults :
290
291 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
292 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
293 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
294 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
295 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
296
297Sometimes a failure to start may happen for whatever reason. Then it is
298important to verify if the version of HAProxy you are invoking is the expected
299version and if it supports the features you are expecting (eg: SSL, PCRE,
300compression, Lua, etc). This can be verified using "haproxy -vv". Some
301important information such as certain build options, the target system and
302the versions of the libraries being used are reported there. It is also what
303you will systematically be asked for when posting a bug report :
304
305 $ haproxy -vv
306 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
307 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
308
309 Build options :
310 TARGET = linux2628
311 CPU = generic
312 CC = gcc
313 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
314 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
315 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
316
317 Default settings :
318 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
319
320 Encrypted password support via crypt(3): yes
321 Built with zlib version : 1.2.6
322 Compression algorithms supported : identity("identity"), deflate("deflate"), \
323 raw-deflate("deflate"), gzip("gzip")
324 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
325 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
326 OpenSSL library supports TLS extensions : yes
327 OpenSSL library supports SNI : yes
328 OpenSSL library supports prefer-server-ciphers : yes
329 Built with PCRE version : 8.12 2011-01-15
330 PCRE library supports JIT : no (USE_PCRE_JIT not set)
331 Built with Lua version : Lua 5.3.1
332 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
333
334 Available polling systems :
335 epoll : pref=300, test result OK
336 poll : pref=200, test result OK
337 select : pref=150, test result OK
338 Total: 3 (3 usable), will use epoll.
339
340The relevant information that many non-developer users can verify here are :
341 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
342 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
343 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
344 fact "1.6-dev7". This is the 7th development version of what will become
345 version 1.6 in the future. A development version not suitable for use in
346 production (unless you know exactly what you are doing). A stable version
347 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
348 14th level of fix on top of version 1.5. This is a production-ready version.
349
350 - the release date : 2015/10/08. It is represented in the universal
351 year/month/day format. Here this means August 8th, 2015. Given that stable
352 releases are issued every few months (1-2 months at the beginning, sometimes
353 6 months once the product becomes very stable), if you're seeing an old date
354 here, it means you're probably affected by a number of bugs or security
355 issues that have since been fixed and that it might be worth checking on the
356 official site.
357
358 - build options : they are relevant to people who build their packages
359 themselves, they can explain why things are not behaving as expected. For
360 example the development version above was built for Linux 2.6.28 or later,
361 targetting a generic CPU (no CPU-specific optimizations), and lacks any
362 code optimization (-O0) so it will perform poorly in terms of performance.
363
364 - libraries versions : zlib version is reported as found in the library
365 itself. In general zlib is considered a very stable product and upgrades
366 are almost never needed. OpenSSL reports two versions, the version used at
367 build time and the one being used, as found on the system. These ones may
368 differ by the last letter but never by the numbers. The build date is also
369 reported because most OpenSSL bugs are security issues and need to be taken
370 seriously, so this library absolutely needs to be kept up to date. Seeing a
371 4-months old version here is highly suspicious and indeed an update was
372 missed. PCRE provides very fast regular expressions and is highly
373 recommended. Certain of its extensions such as JIT are not present in all
374 versions and still young so some people prefer not to build with them,
375 which is why the biuld status is reported as well. Regarding the Lua
376 scripting language, HAProxy expects version 5.3 which is very young since
377 it was released a little time before HAProxy 1.6. It is important to check
378 on the Lua web site if some fixes are proposed for this branch.
379
380 - Available polling systems will affect the process's scalability when
381 dealing with more than about one thousand of concurrent connections. These
382 ones are only available when the correct system was indicated in the TARGET
383 variable during the build. The "epoll" mechanism is highly recommended on
384 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
385 will result in poll() or even select() being used, causing a high CPU usage
386 when dealing with a lot of connections.
387
388
3894. Stopping and restarting HAProxy
390----------------------------------
391
392HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
393SIGTERM signal is sent to the haproxy process, it immediately quits and all
394established connections are closed. The graceful stop is triggered when the
395SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
396from listening ports, but continue to process existing connections until they
397close. Once the last connection is closed, the process leaves.
398
399The hard stop method is used for the "stop" or "restart" actions of the service
400management script. The graceful stop is used for the "reload" action which
401tries to seamlessly reload a new configuration in a new process.
402
403Both of these signals may be sent by the new haproxy process itself during a
404reload or restart, so that they are sent at the latest possible moment and only
405if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
406(graceful) options respectively.
407
408To understand better how these signals are used, it is important to understand
409the whole restart mechanism.
410
411First, an existing haproxy process is running. The administrator uses a system
412specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
413take the new configuration file into effect. What happens then is the following.
414First, the service script (/etc/init.d/haproxy or equivalent) will verify that
415the configuration file parses correctly using "haproxy -c". After that it will
416try to start haproxy with this configuration file, using "-st" or "-sf".
417
418Then HAProxy tries to bind to all listening ports. If some fatal errors happen
419(eg: address not present on the system, permission denied), the process quits
420with an error. If a socket binding fails because a port is already in use, then
421the process will first send a SIGTTOU signal to all the pids specified in the
422"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
423all existing haproxy processes to temporarily stop listening to their ports so
424that the new process can try to bind again. During this time, the old process
425continues to process existing connections. If the binding still fails (because
426for example a port is shared with another daemon), then the new process sends a
427SIGTTIN signal to the old processes to instruct them to resume operations just
428as if nothing happened. The old processes will then restart listening to the
429ports and continue to accept connections. Not that this mechanism is system
430dependant and some operating systems may not support it in multi-process mode.
431
432If the new process manages to bind correctly to all ports, then it sends either
433the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
434of "-sf") to all processes to notify them that it is now in charge of operations
435and that the old processes will have to leave, either immediately or once they
436have finished their job.
437
438It is important to note that during this timeframe, there are two small windows
439of a few milliseconds each where it is possible that a few connection failures
440will be noticed during high loads. Typically observed failure rates are around
4411 failure during a reload operation every 10000 new connections per second,
442which means that a heavily loaded site running at 30000 new connections per
443second may see about 3 failed connection upon every reload. The two situations
444where this happens are :
445
446 - if the new process fails to bind due to the presence of the old process,
447 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
448 typically lasts about one millisecond for a few tens of frontends, and
449 during which some ports will not be bound to the old process and not yet
450 bound to the new one. HAProxy works around this on systems that support the
451 SO_REUSEPORT socket options, as it allows the new process to bind without
452 first asking the old one to unbind. Most BSD systems have been supporting
453 this almost forever. Linux has been supporting this in version 2.0 and
454 dropped it around 2.2, but some patches were floating around by then. It
455 was reintroduced in kernel 3.9, so if you are observing a connection
456 failure rate above the one mentionned above, please ensure that your kernel
457 is 3.9 or newer, or that relevant patches were backported to your kernel
458 (less likely).
459
460 - when the old processes close the listening ports, the kernel may not always
461 redistribute any pending connection that was remaining in the socket's
462 backlog. Under high loads, a SYN packet may happen just before the socket
463 is closed, and will lead to an RST packet being sent to the client. In some
464 critical environments where even one drop is not acceptable, these ones are
465 sometimes dealt with using firewall rules to block SYN packets during the
466 reload, forcing the client to retransmit. This is totally system-dependent,
467 as some systems might be able to visit other listening queues and avoid
468 this RST. A second case concerns the ACK from the client on a local socket
469 that was in SYN_RECV state just before the close. This ACK will lead to an
470 RST packet while the haproxy process is still not aware of it. This one is
471 harder to get rid of, though the firewall filtering rules mentionned above
472 will work well if applied one second or so before restarting the process.
473
474For the vast majority of users, such drops will never ever happen since they
475don't have enough load to trigger the race conditions. And for most high traffic
476users, the failure rate is still fairly within the noise margin provided that at
477least SO_REUSEPORT is properly supported on their systems.
478
479
4805. File-descriptor limitations
481------------------------------
482
483In order to ensure that all incoming connections will successfully be served,
484HAProxy computes at load time the total number of file descriptors that will be
485needed during the process's life. A regular Unix process is generally granted
4861024 file descriptors by default, and a privileged process can raise this limit
487itself. This is one reason for starting HAProxy as root and letting it adjust
488the limit. The default limit of 1024 file descriptors roughly allow about 500
489concurrent connections to be processed. The computation is based on the global
490maxconn parameter which limits the total number of connections per process, the
491number of listeners, the number of servers which have a health check enabled,
492the agent checks, the peers, the loggers and possibly a few other technical
493requirements. A simple rough estimate of this number consists in simply
494doubling the maxconn value and adding a few tens to get the approximate number
495of file descriptors needed.
496
497Originally HAProxy did not know how to compute this value, and it was necessary
498to pass the value using the "ulimit-n" setting in the global section. This
499explains why even today a lot of configurations are seen with this setting
500present. Unfortunately it was often miscalculated resulting in connection
501failures when approaching maxconn instead of throttling incoming connection
502while waiting for the needed resources. For this reason it is important to
503remove any vestigal "ulimit-n" setting that can remain from very old versions.
504
505Raising the number of file descriptors to accept even moderate loads is
506mandatory but comes with some OS-specific adjustments. First, the select()
507polling system is limited to 1024 file descriptors. In fact on Linux it used
508to be capable of handling more but since certain OS ship with excessively
509restrictive SELinux policies forbidding the use of select() with more than
5101024 file descriptors, HAProxy now refuses to start in this case in order to
511avoid any issue at run time. On all supported operating systems, poll() is
512available and will not suffer from this limitation. It is automatically picked
513so there is nothing ot do to get a working configuration. But poll's becomes
514very slow when the number of file descriptors increases. While HAProxy does its
515best to limit this performance impact (eg: via the use of the internal file
516descriptor cache and batched processing), a good rule of thumb is that using
517poll() with more than a thousand concurrent connections will use a lot of CPU.
518
519For Linux systems base on kernels 2.6 and above, the epoll() system call will
520be used. It's a much more scalable mechanism relying on callbacks in the kernel
521that guarantee a constant wake up time regardless of the number of registered
522monitored file descriptors. It is automatically used where detected, provided
523that HAProxy had been built for one of the Linux flavors. Its presence and
524support can be verified using "haproxy -vv".
525
526For BSD systems which support it, kqueue() is available as an alternative. It
527is much faster than poll() and even slightly faster than epoll() thanks to its
528batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
529with Linux's epoll(), its support and availability are reported in the output
530of "haproxy -vv".
531
532Having a good poller is one thing, but it is mandatory that the process can
533reach the limits. When HAProxy starts, it immediately sets the new process's
534file descriptor limits and verifies if it succeeds. In case of failure, it
535reports it before forking so that the administrator can see the problem. As
536long as the process is started by as root, there should be no reason for this
537setting to fail. However, it can fail if the process is started by an
538unprivileged user. If there is a compelling reason for *not* starting haproxy
539as root (eg: started by end users, or by a per-application account), then the
540file descriptor limit can be raised by the system administrator for this
541specific user. The effectiveness of the setting can be verified by issuing
542"ulimit -n" from the user's command line. It should reflect the new limit.
543
544Warning: when an unprivileged user's limits are changed in this user's account,
545it is fairly common that these values are only considered when the user logs in
546and not at all in some scripts run at system boot time nor in crontabs. This is
547totally dependent on the operating system, keep in mind to check "ulimit -n"
548before starting haproxy when running this way. The general advice is never to
549start haproxy as an unprivileged user for production purposes. Another good
550reason is that it prevents haproxy from enabling some security protections.
551
552Once it is certain that the system will allow the haproxy process to use the
553requested number of file descriptors, two new system-specific limits may be
554encountered. The first one is the system-wide file descriptor limit, which is
555the total number of file descriptors opened on the system, covering all
556processes. When this limit is reached, accept() or socket() will typically
557return ENFILE. The second one is the per-process hard limit on the number of
558file descriptors, it prevents setrlimit() from being set higher. Both are very
559dependent on the operating system. On Linux, the system limit is set at boot
560based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
561And the per-process hard limit is set to 1048576 by default, but it can be
562changed using the "fs.nr_open" sysctl.
563
564File descriptor limitations may be observed on a running process when they are
565set too low. The strace utility will report that accept() and socket() return
566"-1 EMFILE" when the process's limits have been reached. In this case, simply
567raising the "ulimit-n" value (or removing it) will solve the problem. If these
568system calls return "-1 ENFILE" then it means that the kernel's limits have
569been reached and that something must be done on a system-wide parameter. These
570trouble must absolutely be addressed, as they result in high CPU usage (when
571accept() fails) and failed connections that are generally visible to the user.
572One solution also consists in lowering the global maxconn value to enforce
573serialization, and possibly to disable HTTP keep-alive to force connections
574to be released and reused faster.
575
576
5776. Memory management
578--------------------
579
580HAProxy uses a simple and fast pool-based memory management. Since it relies on
581a small number of different object types, it's much more efficient to pick new
582objects from a pool which already contains objects of the appropriate size than
583to call malloc() for each different size. The pools are organized as a stack or
584LIFO, so that newly allocated objects are taken from recently released objects
585still hot in the CPU caches. Pools of similar sizes are merged together, in
586order to limit memory fragmentation.
587
588By default, since the focus is set on performance, each released object is put
589back into the pool it came from, and allocated objects are never freed since
590they are expected to be reused very soon.
591
592On the CLI, it is possible to check how memory is being used in pools thanks to
593the "show pools" command :
594
595 > show pools
596 Dumping pools usage. Use SIGQUIT to flush them.
597 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
598 - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
599 - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
600 - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED]
601 - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED]
602 - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
603 - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED]
604 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
605 - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED]
606 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
607 - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED]
608 Total: 11 pools, 26608 bytes allocated, 18544 used.
609
610The pool name is only indicative, it's the name of the first object type using
611this pool. The size in parenthesis is the object size for objects in this pool.
612Object sizes are always rounded up to the closest multiple of 16 bytes. The
613number of objects currently allocated and the equivalent number of bytes is
614reported so that it is easy to know which pool is responsible for the highest
615memory usage. The number of objects currently in use is reported as well in the
616"used" field. The difference between "allocated" and "used" corresponds to the
617objects that have been freed and are available for immediate use.
618
619It is possible to limit the amount of memory allocated per process using the
620"-m" command line option, followed by a number of megabytes. It covers all of
621the process's addressable space, so that includes memory used by some libraries
622as well as the stack, but it is a reliable limit when building a resource
623constrained system. It works the same way as "ulimit -v" on systems which have
624it, or "ulimit -d" for the other ones.
625
626If a memory allocation fails due to the memory limit being reached or because
627the system doesn't have any enough memory, then haproxy will first start to
628free all available objects from all pools before attempting to allocate memory
629again. This mechanism of releasing unused memory can be triggered by sending
630the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
631to the flush will also be reported to stderr when the process runs in
632foreground.
633
634During a reload operation, the process switched to the graceful stop state also
635automatically performs some flushes after releasing any connection so that all
636possible memory is released to save it for the new process.
637
638
6397. CPU usage
640------------
641
642HAProxy normally spends most of its time in the system and a smaller part in
643userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
644connection setups and closes per second at 100% CPU on a single core. When one
645core is saturated, typical figures are :
646 - 95% system, 5% user for long TCP connections or large HTTP objects
647 - 85% system and 15% user for short TCP connections or small HTTP objects in
648 close mode
649 - 70% system and 30% user for small HTTP objects in keep-alive mode
650
651The amount of rules processing and regular expressions will increase the user
652land part. The presence of firewall rules, connection tracking, complex routing
653tables in the system will instead increase the system part.
654
655On most systems, the CPU time observed during network transfers can be cut in 4
656parts :
657 - the interrupt part, which concerns all the processing performed upon I/O
658 receipt, before the target process is even known. Typically Rx packets are
659 accounted for in interrupt. On some systems such as Linux where interrupt
660 processing may be deferred to a dedicated thread, it can appear as softirq,
661 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
662 this load is generally defined by the hardware settings, though in the case
663 of softirq it is often possible to remap the processing to another CPU.
664 This interrupt part will often be perceived as parasitic since it's not
665 associated with any process, but it actually is some processing being done
666 to prepare the work for the process.
667
668 - the system part, which concerns all the processing done using kernel code
669 called from userland. System calls are accounted as system for example. All
670 synchronously delivered Tx packets will be accounted for as system time. If
671 some packets have to be deferred due to queues filling up, they may then be
672 processed in interrupt context later (eg: upon receipt of an ACK opening a
673 TCP window).
674
675 - the user part, which exclusively runs application code in userland. HAProxy
676 runs exclusively in this part, though it makes heavy use of system calls.
677 Rules processing, regular expressions, compression, encryption all add to
678 the user portion of CPU consumption.
679
680 - the idle part, which is what the CPU does when there is nothing to do. For
681 example HAProxy waits for an incoming connection, or waits for some data to
682 leave, meaning the system is waiting for an ACK from the client to push
683 these data.
684
685In practice regarding HAProxy's activity, it is in general reasonably accurate
686(but totally inexact) to consider that interrupt/softirq are caused by Rx
687processing in kernel drivers, that user-land is caused by layer 7 processing
688in HAProxy, and that system time is caused by network processing on the Tx
689path.
690
691Since HAProxy runs around an event loop, it waits for new events using poll()
692(or any alternative) and processes all these events as fast as possible before
693going back to poll() waiting for new events. It measures the time spent waiting
694in poll() compared to the time spent doing processing events. The ratio of
695polling time vs total time is called the "idle" time, it's the amount of time
696spent waiting for something to happen. This ratio is reported in the stats page
697on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
698the load is extremely low. When it's close to 0%, it means that there is
699constantly some activity. While it cannot be very accurate on an overloaded
700system due to other processes possibly preempting the CPU from the haproxy
701process, it still provides a good estimate about how HAProxy considers it is
702working : if the load is low and the idle ratio is low as well, it may indicate
703that HAProxy has a lot of work to do, possibly due to very expensive rules that
704have to be processed. Conversely, if HAProxy indicates the idle is close to
705100% while things are slow, it means that it cannot do anything to speed things
706up because it is already waiting for incoming data to process. In the example
707below, haproxy is completely idle :
708
709 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
710 Idle_pct: 100
711
712When the idle ratio starts to become very low, it is important to tune the
713system and place processes and interrupts correctly to save the most possible
714CPU resources for all tasks. If a firewall is present, it may be worth trying
715to disable it or to tune it to ensure it is not responsible for a large part
716of the performance limitation. It's worth noting that unloading a stateful
717firewall generally reduces both the amount of interrupt/softirq and of system
718usage since such firewalls act both on the Rx and the Tx paths. On Linux,
719unloading the nf_conntrack and ip_conntrack modules will show whether there is
720anything to gain. If so, then the module runs with default settings and you'll
721have to figure how to tune it for better performance. In general this consists
722in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
723disable the "pf" firewall and its stateful engine at the same time.
724
725If it is observed that a lot of time is spent in interrupt/softirq, it is
726important to ensure that they don't run on the same CPU. Most systems tend to
727pin the tasks on the CPU where they receive the network traffic because for
728certain workloads it improves things. But with heavily network-bound workloads
729it is the opposite as the haproxy process will have to fight against its kernel
730counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
731all sharing the same L3 cache tends to sensibly increase network performance
732because in practice the amount of work for haproxy and the network stack are
733quite close, so they can almost fill an entire CPU each. On Linux this is done
734using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
735interrupts are assigned under /proc/irq. Many network interfaces support
736multiple queues and multiple interrupts. In general it helps to spread them
737across a small number of CPU cores provided they all share the same L3 cache.
738Please always stop irq_balance which always does the worst possible thing on
739such workloads.
740
741For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
742compression, it may be worth using multiple processes dedicated to certain
743tasks, though there is no universal rule here and experimentation will have to
744be performed.
745
746In order to increase the CPU capacity, it is possible to make HAProxy run as
747several processes, using the "nbproc" directive in the global section. There
748are some limitations though :
749 - health checks are run per process, so the target servers will get as many
750 checks as there are running processes ;
751 - maxconn values and queues are per-process so the correct value must be set
752 to avoid overloading the servers ;
753 - outgoing connections should avoid using port ranges to avoid conflicts
754 - stick-tables are per process and are not shared between processes ;
755 - each peers section may only run on a single process at a time ;
756 - the CLI operations will only act on a single process at a time.
757
758With this in mind, it appears that the easiest setup often consists in having
759one first layer running on multiple processes and in charge for the heavy
760processing, passing the traffic to a second layer running in a single process.
761This mechanism is suited to SSL and compression which are the two CPU-heavy
762features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800763than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200764useful to pass client information to the next stage. When doing so, it is
765generally a good idea to bind all the single-process tasks to process number 1
766and extra tasks to next processes, as this will make it easier to generate
767similar configurations for different machines.
768
769On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
770more efficient when each process uses a distinct listening socket on the same
771IP:port ; this will make the kernel evenly distribute the load across all
772processes instead of waking them all up. Please check the "process" option of
773the "bind" keyword lines in the configuration manual for more information.
774
775
7768. Logging
777----------
778
779For logging, HAProxy always relies on a syslog server since it does not perform
780any file-system access. The standard way of using it is to send logs over UDP
781to the log server (by default on port 514). Very commonly this is configured to
782127.0.0.1 where the local syslog daemon is running, but it's also used over the
783network to log to a central server. The central server provides additional
784benefits especially in active-active scenarios where it is desirable to keep
785the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
786send its logs to the local syslog daemon, but it is not recommended at all,
787because if the syslog server is restarted while haproxy runs, the socket will
788be replaced and new logs will be lost. Since HAProxy will be isolated inside a
789chroot jail, it will not have the ability to reconnect to the new socket. It
790has also been observed in field that the log buffers in use on UNIX sockets are
791very small and lead to lost messages even at very light loads. But this can be
792fine for testing however.
793
794It is recommended to add the following directive to the "global" section to
795make HAProxy log to the local daemon using facility "local0" :
796
797 log 127.0.0.1:514 local0
798
799and then to add the following one to each "defaults" section or to each frontend
800and backend section :
801
802 log global
803
804This way, all logs will be centralized through the global definition of where
805the log server is.
806
807Some syslog daemons do not listen to UDP traffic by default, so depending on
808the daemon being used, the syntax to enable this will vary :
809
810 - on sysklogd, you need to pass argument "-r" on the daemon's command line
811 so that it listens to a UDP socket for "remote" logs ; note that there is
812 no way to limit it to address 127.0.0.1 so it will also receive logs from
813 remote systems ;
814
815 - on rsyslogd, the following lines must be added to the configuration file :
816
817 $ModLoad imudp
818 $UDPServerAddress *
819 $UDPServerRun 514
820
821 - on syslog-ng, a new source can be created the following way, it then needs
822 to be added as a valid source in one of the "log" directives :
823
824 source s_udp {
825 udp(ip(127.0.0.1) port(514));
826 };
827
828Please consult your syslog daemon's manual for more information. If no logs are
829seen in the system's log files, please consider the following tests :
830
831 - restart haproxy. Each frontend and backend logs one line indicating it's
832 starting. If these logs are received, it means logs are working.
833
834 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
835 activity that you expect to be logged. You should see the log messages
836 being sent using sendmsg() there. If they don't appear, restart using
837 strace on top of haproxy. If you still see no logs, it definitely means
838 that something is wrong in your configuration.
839
840 - run tcpdump to watch for port 514, for example on the loopback interface if
841 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
842 packets are seen there, it's the proof they're sent then the syslogd daemon
843 needs to be troubleshooted.
844
845While traffic logs are sent from the frontends (where the incoming connections
846are accepted), backends also need to be able to send logs in order to report a
847server state change consecutive to a health check. Please consult HAProxy's
848configuration manual for more information regarding all possible log settings.
849
850It is convenient to chose a facility that is not used by other deamons. HAProxy
851examples often suggest "local0" for traffic logs and "local1" for admin logs
852because they're never seen in field. A single facility would be enough as well.
853Having separate logs is convenient for log analysis, but it's also important to
854remember that logs may sometimes convey confidential information, and as such
855they must not be mixed with other logs that may accidently be handed out to
856unauthorized people.
857
858For in-field troubleshooting without impacting the server's capacity too much,
859it is recommended to make use of the "halog" utility provided with HAProxy.
860This is sort of a grep-like utility designed to process HAProxy log files at
861a very fast data rate. Typical figures range between 1 and 2 GB of logs per
862second. It is capable of extracting only certain logs (eg: search for some
863classes of HTTP status codes, connection termination status, search by response
864time ranges, look for errors only), count lines, limit the output to a number
865of lines, and perform some more advanced statistics such as sorting servers
866by response time or error counts, sorting URLs by time or count, sorting client
867addresses by access count, and so on. It is pretty convenient to quickly spot
868anomalies such as a bot looping on the site, and block them.
869
870
8719. Statistics and monitoring
872----------------------------
873
Willy Tarreau44aed902015-10-13 14:45:29 +0200874It is possible to query HAProxy about its status. The most commonly used
875mechanism is the HTTP statistics page. This page also exposes an alternative
876CSV output format for monitoring tools. The same format is provided on the
877Unix socket.
878
879
8809.1. CSV format
881---------------
882
883The statistics may be consulted either from the unix socket or from the HTTP
884page. Both means provide a CSV format whose fields follow. The first line
885begins with a sharp ('#') and has one word per comma-delimited field which
886represents the title of the column. All other lines starting at the second one
887use a classical CSV format using a comma as the delimiter, and the double quote
888('"') as an optional text delimiter, but only if the enclosed text is ambiguous
889(if it contains a quote or a comma). The double-quote character ('"') in the
890text is doubled ('""'), which is the format that most tools recognize. Please
891do not insert any column before these ones in order not to break tools which
892use hard-coded column positions.
893
894In brackets after each field name are the types which may have a value for
895that field. The types are L (Listeners), F (Frontends), B (Backends), and
896S (Servers).
897
898 0. pxname [LFBS]: proxy name
899 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
900 any name for server/listener)
901 2. qcur [..BS]: current queued requests. For the backend this reports the
902 number queued without a server assigned.
903 3. qmax [..BS]: max value of qcur
904 4. scur [LFBS]: current sessions
905 5. smax [LFBS]: max sessions
906 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100907 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200908 8. bin [LFBS]: bytes in
909 9. bout [LFBS]: bytes out
910 10. dreq [LFB.]: requests denied because of security concerns.
911 - For tcp this is because of a matched tcp-request content rule.
912 - For http this is because of a matched http-request or tarpit rule.
913 11. dresp [LFBS]: responses denied because of security concerns.
914 - For http this is because of a matched http-request rule, or
915 "option checkcache".
916 12. ereq [LF..]: request errors. Some of the possible causes are:
917 - early termination from the client, before the request has been sent.
918 - read error from the client
919 - client timeout
920 - client closed connection
921 - various bad requests from the client.
922 - request was tarpitted.
923 13. econ [..BS]: number of requests that encountered an error trying to
924 connect to a backend server. The backend stat is the sum of the stat
925 for all servers of that backend, plus any connection errors not
926 associated with a particular server (such as the backend having no
927 active servers).
928 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
929 Some other errors are:
930 - write error on the client socket (won't be counted for the server stat)
931 - failure applying filters to the response.
932 15. wretr [..BS]: number of times a connection to a server was retried.
933 16. wredis [..BS]: number of times a request was redispatched to another
934 server. The server value counts the number of times that server was
935 switched away from.
936 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)...)
937 18. weight [..BS]: total weight (backend), server weight (server)
938 19. act [..BS]: number of active servers (backend), server is active (server)
939 20. bck [..BS]: number of backup servers (backend), server is backup (server)
940 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
941 the server is up.)
942 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
943 transitions to the whole backend being down, rather than the sum of the
944 counters for each server.
945 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
946 24. downtime [..BS]: total downtime (in seconds). The value for the backend
947 is the downtime for the whole backend, not the sum of the server downtime.
948 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
949 value is 0 (default, meaning no limit)
950 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
951 27. iid [LFBS]: unique proxy id
952 28. sid [L..S]: server id (unique inside a proxy)
953 29. throttle [...S]: current throttle percentage for the server, when
954 slowstart is active, or no value if not in slowstart.
955 30. lbtot [..BS]: total number of times a server was selected, either for new
956 sessions, or when re-dispatching. The server counter is the number
957 of times that server was selected.
958 31. tracked [...S]: id of proxy/server if tracking is enabled.
959 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
960 33. rate [.FBS]: number of sessions per second over last elapsed second
961 34. rate_lim [.F..]: configured limit on new sessions per second
962 35. rate_max [.FBS]: max number of new sessions per second
963 36. check_status [...S]: status of last health check, one of:
964 UNK -> unknown
965 INI -> initializing
966 SOCKERR -> socket error
967 L4OK -> check passed on layer 4, no upper layers testing enabled
968 L4TOUT -> layer 1-4 timeout
969 L4CON -> layer 1-4 connection problem, for example
970 "Connection refused" (tcp rst) or "No route to host" (icmp)
971 L6OK -> check passed on layer 6
972 L6TOUT -> layer 6 (SSL) timeout
973 L6RSP -> layer 6 invalid response - protocol error
974 L7OK -> check passed on layer 7
975 L7OKC -> check conditionally passed on layer 7, for example 404 with
976 disable-on-404
977 L7TOUT -> layer 7 (HTTP/SMTP) timeout
978 L7RSP -> layer 7 invalid response - protocol error
979 L7STS -> layer 7 response error, for example HTTP 5xx
980 37. check_code [...S]: layer5-7 code, if available
981 38. check_duration [...S]: time in ms took to finish last health check
982 39. hrsp_1xx [.FBS]: http responses with 1xx code
983 40. hrsp_2xx [.FBS]: http responses with 2xx code
984 41. hrsp_3xx [.FBS]: http responses with 3xx code
985 42. hrsp_4xx [.FBS]: http responses with 4xx code
986 43. hrsp_5xx [.FBS]: http responses with 5xx code
987 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
988 45. hanafail [...S]: failed health checks details
989 46. req_rate [.F..]: HTTP requests per second over last elapsed second
990 47. req_rate_max [.F..]: max number of HTTP requests per second observed
991 48. req_tot [.F..]: total number of HTTP requests received
992 49. cli_abrt [..BS]: number of data transfers aborted by the client
993 50. srv_abrt [..BS]: number of data transfers aborted by the server
994 (inc. in eresp)
995 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
996 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
997 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
998 (CPU/BW limit)
999 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1000 55. lastsess [..BS]: number of seconds since last session assigned to
1001 server/backend
1002 56. last_chk [...S]: last health check contents or textual error
1003 57. last_agt [...S]: last agent check contents or textual error
1004 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1005 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1006 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1007 (0 for TCP)
1008 61. ttime [..BS]: the average total session time in ms over the 1024 last
1009 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001010 62. agent_status [...S]: status of last agent check, one of:
1011 UNK -> unknown
1012 INI -> initializing
1013 SOCKERR -> socket error
1014 L4OK -> check passed on layer 4, no upper layers testing enabled
1015 L4TOUT -> layer 1-4 timeout
1016 L4CON -> layer 1-4 connection problem, for example
1017 "Connection refused" (tcp rst) or "No route to host" (icmp)
1018 L7OK -> agent reported "up"
1019 L7STS -> agent reported "fail", "stop", or "down"
1020 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1021 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001022 65. check_desc [...S]: short human-readable description of check_status
1023 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001024 67. check_rise [...S]: server's "rise" parameter used by checks
1025 68. check_fall [...S]: server's "fall" parameter used by checks
1026 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1027 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1028 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1029 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001030 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001031 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001032 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001033 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001034 77: conn_rate [.F..]: number of connections over the last elapsed second
1035 78: conn_rate_max [.F..]: highest known conn_rate
1036 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001037 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau44aed902015-10-13 14:45:29 +02001038
1039
Willy Tarreau5d8b9792016-03-11 11:09:34 +010010409.2) Typed output format
1041------------------------
1042
1043Both "show info" and "show stat" support a mode where each output value comes
1044with its type and sufficient information to know how the value is supposed to
1045be aggregated between processes and how it evolves.
1046
1047In all cases, the output consists in having a single value per line with all
1048the information split into fields delimited by colons (':').
1049
1050The first column designates the object or metric being dumped. Its format is
1051specific to the command producing this output and will not be described in this
1052section. Usually it will consist in a series of identifiers and field names.
1053
1054The second column contains 3 characters respectively indicating the origin, the
1055nature and the scope of the value being reported. The first character (the
1056origin) indicates where the value was extracted from. Possible characters are :
1057
1058 M The value is a metric. It is valid at one instant any may change depending
1059 on its nature .
1060
1061 S The value is a status. It represents a discrete value which by definition
1062 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1063 the PID of the process, etc.
1064
1065 K The value is a sorting key. It represents an identifier which may be used
1066 to group some values together because it is unique among its class. All
1067 internal identifiers are keys. Some names can be listed as keys if they
1068 are unique (eg: a frontend name is unique). In general keys come from the
1069 configuration, eventhough some of them may automatically be assigned. For
1070 most purposes keys may be considered as equivalent to configuration.
1071
1072 C The value comes from the configuration. Certain configuration values make
1073 sense on the output, for example a concurrent connection limit or a cookie
1074 name. By definition these values are the same in all processes started
1075 from the same configuration file.
1076
1077 P The value comes from the product itself. There are very few such values,
1078 most common use is to report the product name, version and release date.
1079 These elements are also the same between all processes.
1080
1081The second character (the nature) indicates the nature of the information
1082carried by the field in order to let an aggregator decide on what operation to
1083use to aggregate multiple values. Possible characters are :
1084
1085 A The value represents an age since a last event. This is a bit different
1086 from the duration in that an age is automatically computed based on the
1087 current date. A typical example is how long ago did the last session
1088 happen on a server. Ages are generally aggregated by taking the minimum
1089 value and do not need to be stored.
1090
1091 a The value represents an already averaged value. The average response times
1092 and server weights are of this nature. Averages can typically be averaged
1093 between processes.
1094
1095 C The value represents a cumulative counter. Such measures perpetually
1096 increase until they wrap around. Some monitoring protocols need to tell
1097 the difference between a counter and a gauge to report a different type.
1098 In general counters may simply be summed since they represent events or
1099 volumes. Examples of metrics of this nature are connection counts or byte
1100 counts.
1101
1102 D The value represents a duration for a status. There are a few usages of
1103 this, most of them include the time taken by the last health check and
1104 the time a server has spent down. Durations are generally not summed,
1105 most of the time the maximum will be retained to compute an SLA.
1106
1107 G The value represents a gauge. It's a measure at one instant. The memory
1108 usage or the current number of active connections are of this nature.
1109 Metrics of this type are typically summed during aggregation.
1110
1111 L The value represents a limit (generally a configured one). By nature,
1112 limits are harder to aggregate since they are specific to the point where
1113 they were retrieved. In certain situations they may be summed or be kept
1114 separate.
1115
1116 M The value represents a maximum. In general it will apply to a gauge and
1117 keep the highest known value. An example of such a metric could be the
1118 maximum amount of concurrent connections that was encountered in the
1119 product's life time. To correctly aggregate maxima, you are supposed to
1120 output a range going from the maximum of all maxima and the sum of all
1121 of them. There is indeed no way to know if they were encountered
1122 simultaneously or not.
1123
1124 m The value represents a minimum. In general it will apply to a gauge and
1125 keep the lowest known value. An example of such a metric could be the
1126 minimum amount of free memory pools that was encountered in the product's
1127 life time. To correctly aggregate minima, you are supposed to output a
1128 range going from the minimum of all minima and the sum of all of them.
1129 There is indeed no way to know if they were encountered simultaneously
1130 or not.
1131
1132 N The value represents a name, so it is a string. It is used to report
1133 proxy names, server names and cookie names. Names have configuration or
1134 keys as their origin and are supposed to be the same among all processes.
1135
1136 O The value represents a free text output. Outputs from various commands,
1137 returns from health checks, node descriptions are of such nature.
1138
1139 R The value represents an event rate. It's a measure at one instant. It is
1140 quite similar to a gauge except that the recipient knows that this measure
1141 moves slowly and may decide not to keep all values. An example of such a
1142 metric is the measured amount of connections per second. Metrics of this
1143 type are typically summed during aggregation.
1144
1145 T The value represents a date or time. A field emitting the current date
1146 would be of this type. The method to aggregate such information is left
1147 as an implementation choice. For now no field uses this type.
1148
1149The third character (the scope) indicates what extent the value reflects. Some
1150elements may be per process while others may be per configuration or per system.
1151The distinction is important to know whether or not a single value should be
1152kept during aggregation or if values have to be aggregated. The following
1153characters are currently supported :
1154
1155 C The value is valid for a whole cluster of nodes, which is the set of nodes
1156 communicating over the peers protocol. An example could be the amount of
1157 entries present in a stick table that is replicated with other peers. At
1158 the moment no metric use this scope.
1159
1160 P The value is valid only for the process reporting it. Most metrics use
1161 this scope.
1162
1163 S The value is valid for the whole service, which is the set of processes
1164 started together from the same configuration file. All metrics originating
1165 from the configuration use this scope. Some other metrics may use it as
1166 well for some shared resources (eg: shared SSL cache statistics).
1167
1168 s The value is valid for the whole system, such as the system's hostname,
1169 current date or resource usage. At the moment this scope is not used by
1170 any metric.
1171
1172Consumers of these information will generally have enough of these 3 characters
1173to determine how to accurately report aggregated information across multiple
1174processes.
1175
1176After this column, the third column indicates the type of the field, among "s32"
1177(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1178integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1179know the type before parsing the value in order to properly read it. For example
1180a string containing only digits is still a string an not an integer (eg: an
1181error code extracted by a check).
1182
1183Then the fourth column is the value itself, encoded according to its type.
1184Strings are dumped as-is immediately after the colon without any leading space.
1185If a string contains a colon, it will appear normally. This means that the
1186output should not be exclusively split around colons or some check outputs
1187or server addresses might be truncated.
1188
1189
11909.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001191-------------------------
1192
1193The stats socket is not enabled by default. In order to enable it, it is
1194necessary to add one line in the global section of the haproxy configuration.
1195A second line is recommended to set a larger timeout, always appreciated when
1196issuing commands by hand :
1197
1198 global
1199 stats socket /var/run/haproxy.sock mode 600 level admin
1200 stats timeout 2m
1201
1202It is also possible to add multiple instances of the stats socket by repeating
1203the line, and make them listen to a TCP port instead of a UNIX socket. This is
1204never done by default because this is dangerous, but can be handy in some
1205situations :
1206
1207 global
1208 stats socket /var/run/haproxy.sock mode 600 level admin
1209 stats socket ipv4@192.168.0.1:9999 level admin
1210 stats timeout 2m
1211
1212To access the socket, an external utility such as "socat" is required. Socat is
1213a swiss-army knife to connect anything to anything. We use it to connect
1214terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1215The two main syntaxes we'll use are the following :
1216
1217 # socat /var/run/haproxy.sock stdio
1218 # socat /var/run/haproxy.sock readline
1219
1220The first one is used with scripts. It is possible to send the output of a
1221script to haproxy, and pass haproxy's output to another script. That's useful
1222for retrieving counters or attack traces for example.
1223
1224The second one is only useful for issuing commands by hand. It has the benefit
1225that the terminal is handled by the readline library which supports line
1226editing and history, which is very convenient when issuing repeated commands
1227(eg: watch a counter).
1228
1229The socket supports two operation modes :
1230 - interactive
1231 - non-interactive
1232
1233The non-interactive mode is the default when socat connects to the socket. In
1234this mode, a single line may be sent. It is processed as a whole, responses are
1235sent back, and the connection closes after the end of the response. This is the
1236mode that scripts and monitoring tools use. It is possible to send multiple
1237commands in this mode, they need to be delimited by a semi-colon (';'). For
1238example :
1239
1240 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1241
1242The interactive mode displays a prompt ('>') and waits for commands to be
1243entered on the line, then processes them, and displays the prompt again to wait
1244for a new command. This mode is entered via the "prompt" command which must be
1245sent on the first line in non-interactive mode. The mode is a flip switch, if
1246"prompt" is sent in interactive mode, it is disabled and the connection closes
1247after processing the last command of the same line.
1248
1249For this reason, when debugging by hand, it's quite common to start with the
1250"prompt" command :
1251
1252 # socat /var/run/haproxy readline
1253 prompt
1254 > show info
1255 ...
1256 >
1257
1258Since multiple commands may be issued at once, haproxy uses the empty line as a
1259delimiter to mark an end of output for each command, and takes care of ensuring
1260that no command can emit an empty line on output. A script can thus easily
1261parse the output even when multiple commands were pipelined on a single line.
1262
1263It is important to understand that when multiple haproxy processes are started
1264on the same sockets, any process may pick up the request and will output its
1265own stats.
1266
1267The list of commands currently supported on the stats socket is provided below.
1268If an unknown command is sent, haproxy displays the usage message which reminds
1269all supported commands. Some commands support a more complex syntax, generally
1270it will explain what part of the command is invalid when this happens.
1271
1272add acl <acl> <pattern>
1273 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1274 "show acl". This command does not verify if the entry already exists. This
1275 command cannot be used if the reference <acl> is a file also used with a map.
1276 In this case, you must use the command "add map" in place of "add acl".
1277
1278add map <map> <key> <value>
1279 Add an entry into the map <map> to associate the value <value> to the key
1280 <key>. This command does not verify if the entry already exists. It is
1281 mainly used to fill a map after a clear operation. Note that if the reference
1282 <map> is a file and is shared with a map, this map will contain also a new
1283 pattern entry.
1284
1285clear counters
1286 Clear the max values of the statistics counters in each proxy (frontend &
1287 backend) and in each server. The cumulated counters are not affected. This
1288 can be used to get clean counters after an incident, without having to
1289 restart nor to clear traffic counters. This command is restricted and can
1290 only be issued on sockets configured for levels "operator" or "admin".
1291
1292clear counters all
1293 Clear all statistics counters in each proxy (frontend & backend) and in each
1294 server. This has the same effect as restarting. This command is restricted
1295 and can only be issued on sockets configured for level "admin".
1296
1297clear acl <acl>
1298 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1299 returned by "show acl". Note that if the reference <acl> is a file and is
1300 shared with a map, this map will be also cleared.
1301
1302clear map <map>
1303 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1304 returned by "show map". Note that if the reference <map> is a file and is
1305 shared with a acl, this acl will be also cleared.
1306
1307clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1308 Remove entries from the stick-table <table>.
1309
1310 This is typically used to unblock some users complaining they have been
1311 abusively denied access to a service, but this can also be used to clear some
1312 stickiness entries matching a server that is going to be replaced (see "show
1313 table" below for details). Note that sometimes, removal of an entry will be
1314 refused because it is currently tracked by a session. Retrying a few seconds
1315 later after the session ends is usual enough.
1316
1317 In the case where no options arguments are given all entries will be removed.
1318
1319 When the "data." form is used entries matching a filter applied using the
1320 stored data (see "stick-table" in section 4.2) are removed. A stored data
1321 type must be specified in <type>, and this data type must be stored in the
1322 table otherwise an error is reported. The data is compared according to
1323 <operator> with the 64-bit integer <value>. Operators are the same as with
1324 the ACLs :
1325
1326 - eq : match entries whose data is equal to this value
1327 - ne : match entries whose data is not equal to this value
1328 - le : match entries whose data is less than or equal to this value
1329 - ge : match entries whose data is greater than or equal to this value
1330 - lt : match entries whose data is less than this value
1331 - gt : match entries whose data is greater than this value
1332
1333 When the key form is used the entry <key> is removed. The key must be of the
1334 same type as the table, which currently is limited to IPv4, IPv6, integer and
1335 string.
1336
1337 Example :
1338 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1339 >>> # table: http_proxy, type: ip, size:204800, used:2
1340 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1341 bytes_out_rate(60000)=187
1342 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1343 bytes_out_rate(60000)=191
1344
1345 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1346
1347 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1348 >>> # table: http_proxy, type: ip, size:204800, used:1
1349 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1350 bytes_out_rate(60000)=191
1351 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1352 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1353 >>> # table: http_proxy, type: ip, size:204800, used:1
1354
1355del acl <acl> [<key>|#<ref>]
1356 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1357 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1358 this command delete only the listed reference. The reference can be found with
1359 listing the content of the acl. Note that if the reference <acl> is a file and
1360 is shared with a map, the entry will be also deleted in the map.
1361
1362del map <map> [<key>|#<ref>]
1363 Delete all the map entries from the map <map> corresponding to the key <key>.
1364 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1365 this command delete only the listed reference. The reference can be found with
1366 listing the content of the map. Note that if the reference <map> is a file and
1367 is shared with a acl, the entry will be also deleted in the map.
1368
1369disable agent <backend>/<server>
1370 Mark the auxiliary agent check as temporarily stopped.
1371
1372 In the case where an agent check is being run as a auxiliary check, due
1373 to the agent-check parameter of a server directive, new checks are only
1374 initialised when the agent is in the enabled. Thus, disable agent will
1375 prevent any new agent checks from begin initiated until the agent
1376 re-enabled using enable agent.
1377
1378 When an agent is disabled the processing of an auxiliary agent check that
1379 was initiated while the agent was set as enabled is as follows: All
1380 results that would alter the weight, specifically "drain" or a weight
1381 returned by the agent, are ignored. The processing of agent check is
1382 otherwise unchanged.
1383
1384 The motivation for this feature is to allow the weight changing effects
1385 of the agent checks to be paused to allow the weight of a server to be
1386 configured using set weight without being overridden by the agent.
1387
1388 This command is restricted and can only be issued on sockets configured for
1389 level "admin".
1390
1391disable frontend <frontend>
1392 Mark the frontend as temporarily stopped. This corresponds to the mode which
1393 is used during a soft restart : the frontend releases the port but can be
1394 enabled again if needed. This should be used with care as some non-Linux OSes
1395 are unable to enable it back. This is intended to be used in environments
1396 where stopping a proxy is not even imaginable but a misconfigured proxy must
1397 be fixed. That way it's possible to release the port and bind it into another
1398 process to restore operations. The frontend will appear with status "STOP"
1399 on the stats page.
1400
1401 The frontend may be specified either by its name or by its numeric ID,
1402 prefixed with a sharp ('#').
1403
1404 This command is restricted and can only be issued on sockets configured for
1405 level "admin".
1406
1407disable health <backend>/<server>
1408 Mark the primary health check as temporarily stopped. This will disable
1409 sending of health checks, and the last health check result will be ignored.
1410 The server will be in unchecked state and considered UP unless an auxiliary
1411 agent check forces it down.
1412
1413 This command is restricted and can only be issued on sockets configured for
1414 level "admin".
1415
1416disable server <backend>/<server>
1417 Mark the server DOWN for maintenance. In this mode, no more checks will be
1418 performed on the server until it leaves maintenance.
1419 If the server is tracked by other servers, those servers will be set to DOWN
1420 during the maintenance.
1421
1422 In the statistics page, a server DOWN for maintenance will appear with a
1423 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1424
1425 Both the backend and the server may be specified either by their name or by
1426 their numeric ID, prefixed with a sharp ('#').
1427
1428 This command is restricted and can only be issued on sockets configured for
1429 level "admin".
1430
1431enable agent <backend>/<server>
1432 Resume auxiliary agent check that was temporarily stopped.
1433
1434 See "disable agent" for details of the effect of temporarily starting
1435 and stopping an auxiliary agent.
1436
1437 This command is restricted and can only be issued on sockets configured for
1438 level "admin".
1439
1440enable frontend <frontend>
1441 Resume a frontend which was temporarily stopped. It is possible that some of
1442 the listening ports won't be able to bind anymore (eg: if another process
1443 took them since the 'disable frontend' operation). If this happens, an error
1444 is displayed. Some operating systems might not be able to resume a frontend
1445 which was disabled.
1446
1447 The frontend may be specified either by its name or by its numeric ID,
1448 prefixed with a sharp ('#').
1449
1450 This command is restricted and can only be issued on sockets configured for
1451 level "admin".
1452
1453enable health <backend>/<server>
1454 Resume a primary health check that was temporarily stopped. This will enable
1455 sending of health checks again. Please see "disable health" for details.
1456
1457 This command is restricted and can only be issued on sockets configured for
1458 level "admin".
1459
1460enable server <backend>/<server>
1461 If the server was previously marked as DOWN for maintenance, this marks the
1462 server UP and checks are re-enabled.
1463
1464 Both the backend and the server may be specified either by their name or by
1465 their numeric ID, prefixed with a sharp ('#').
1466
1467 This command is restricted and can only be issued on sockets configured for
1468 level "admin".
1469
1470get map <map> <value>
1471get acl <acl> <value>
1472 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1473 are the #<id> or the <file> returned by "show map" or "show acl". This command
1474 returns all the matching patterns associated with this map. This is useful for
1475 debugging maps and ACLs. The output format is composed by one line par
1476 matching type. Each line is composed by space-delimited series of words.
1477
1478 The first two words are:
1479
1480 <match method>: The match method applied. It can be "found", "bool",
1481 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1482 "dom", "end" or "reg".
1483
1484 <match result>: The result. Can be "match" or "no-match".
1485
1486 The following words are returned only if the pattern matches an entry.
1487
1488 <index type>: "tree" or "list". The internal lookup algorithm.
1489
1490 <case>: "case-insensitive" or "case-sensitive". The
1491 interpretation of the case.
1492
1493 <entry matched>: match="<entry>". Return the matched pattern. It is
1494 useful with regular expressions.
1495
1496 The two last word are used to show the returned value and its type. With the
1497 "acl" case, the pattern doesn't exist.
1498
1499 return=nothing: No return because there are no "map".
1500 return="<value>": The value returned in the string format.
1501 return=cannot-display: The value cannot be converted as string.
1502
1503 type="<type>": The type of the returned sample.
1504
1505get weight <backend>/<server>
1506 Report the current weight and the initial weight of server <server> in
1507 backend <backend> or an error if either doesn't exist. The initial weight is
1508 the one that appears in the configuration file. Both are normally equal
1509 unless the current weight has been changed. Both the backend and the server
1510 may be specified either by their name or by their numeric ID, prefixed with a
1511 sharp ('#').
1512
1513help
1514 Print the list of known keywords and their basic usage. The same help screen
1515 is also displayed for unknown commands.
1516
1517prompt
1518 Toggle the prompt at the beginning of the line and enter or leave interactive
1519 mode. In interactive mode, the connection is not closed after a command
1520 completes. Instead, the prompt will appear again, indicating the user that
1521 the interpreter is waiting for a new command. The prompt consists in a right
1522 angle bracket followed by a space "> ". This mode is particularly convenient
1523 when one wants to periodically check information such as stats or errors.
1524 It is also a good idea to enter interactive mode before issuing a "help"
1525 command.
1526
1527quit
1528 Close the connection when in interactive mode.
1529
1530set map <map> [<key>|#<ref>] <value>
1531 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1532 #<id> or <file> returned by "show map". If the <ref> is used in place of
1533 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1534
1535set maxconn frontend <frontend> <value>
1536 Dynamically change the specified frontend's maxconn setting. Any positive
1537 value is allowed including zero, but setting values larger than the global
1538 maxconn does not make much sense. If the limit is increased and connections
1539 were pending, they will immediately be accepted. If it is lowered to a value
1540 below the current number of connections, new connections acceptation will be
1541 delayed until the threshold is reached. The frontend might be specified by
1542 either its name or its numeric ID prefixed with a sharp ('#').
1543
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001544set maxconn server <backend/server> <value>
1545 Dynamically change the specified server's maxconn setting. Any positive
1546 value is allowed including zero, but setting values larger than the global
1547 maxconn does not make much sense.
1548
Willy Tarreau44aed902015-10-13 14:45:29 +02001549set maxconn global <maxconn>
1550 Dynamically change the global maxconn setting within the range defined by the
1551 initial global maxconn setting. If it is increased and connections were
1552 pending, they will immediately be accepted. If it is lowered to a value below
1553 the current number of connections, new connections acceptation will be
1554 delayed until the threshold is reached. A value of zero restores the initial
1555 setting.
1556
1557set rate-limit connections global <value>
1558 Change the process-wide connection rate limit, which is set by the global
1559 'maxconnrate' setting. A value of zero disables the limitation. This limit
1560 applies to all frontends and the change has an immediate effect. The value
1561 is passed in number of connections per second.
1562
1563set rate-limit http-compression global <value>
1564 Change the maximum input compression rate, which is set by the global
1565 'maxcomprate' setting. A value of zero disables the limitation. The value is
1566 passed in number of kilobytes per second. The value is available in the "show
1567 info" on the line "CompressBpsRateLim" in bytes.
1568
1569set rate-limit sessions global <value>
1570 Change the process-wide session rate limit, which is set by the global
1571 'maxsessrate' setting. A value of zero disables the limitation. This limit
1572 applies to all frontends and the change has an immediate effect. The value
1573 is passed in number of sessions per second.
1574
1575set rate-limit ssl-sessions global <value>
1576 Change the process-wide SSL session rate limit, which is set by the global
1577 'maxsslrate' setting. A value of zero disables the limitation. This limit
1578 applies to all frontends and the change has an immediate effect. The value
1579 is passed in number of sessions per second sent to the SSL stack. It applies
1580 before the handshake in order to protect the stack against handshake abuses.
1581
1582set server <backend>/<server> addr <ip4 or ip6 address>
1583 Replace the current IP address of a server by the one provided.
1584
1585set server <backend>/<server> agent [ up | down ]
1586 Force a server's agent to a new state. This can be useful to immediately
1587 switch a server's state regardless of some slow agent checks for example.
1588 Note that the change is propagated to tracking servers if any.
1589
1590set server <backend>/<server> health [ up | stopping | down ]
1591 Force a server's health to a new state. This can be useful to immediately
1592 switch a server's state regardless of some slow health checks for example.
1593 Note that the change is propagated to tracking servers if any.
1594
1595set server <backend>/<server> state [ ready | drain | maint ]
1596 Force a server's administrative state to a new state. This can be useful to
1597 disable load balancing and/or any traffic to a server. Setting the state to
1598 "ready" puts the server in normal mode, and the command is the equivalent of
1599 the "enable server" command. Setting the state to "maint" disables any traffic
1600 to the server as well as any health checks. This is the equivalent of the
1601 "disable server" command. Setting the mode to "drain" only removes the server
1602 from load balancing but still allows it to be checked and to accept new
1603 persistent connections. Changes are propagated to tracking servers if any.
1604
1605set server <backend>/<server> weight <weight>[%]
1606 Change a server's weight to the value passed in argument. This is the exact
1607 equivalent of the "set weight" command below.
1608
1609set ssl ocsp-response <response>
1610 This command is used to update an OCSP Response for a certificate (see "crt"
1611 on "bind" lines). Same controls are performed as during the initial loading of
1612 the response. The <response> must be passed as a base64 encoded string of the
1613 DER encoded response from the OCSP server.
1614
1615 Example:
1616 openssl ocsp -issuer issuer.pem -cert server.pem \
1617 -host ocsp.issuer.com:80 -respout resp.der
1618 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1619 socat stdio /var/run/haproxy.stat
1620
1621set ssl tls-key <id> <tlskey>
1622 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1623 ultimate key, while the penultimate one is used for encryption (others just
1624 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1625 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
1626 bit TLS ticket key (ex. openssl rand -base64 48).
1627
1628set table <table> key <key> [data.<data_type> <value>]*
1629 Create or update a stick-table entry in the table. If the key is not present,
1630 an entry is inserted. See stick-table in section 4.2 to find all possible
1631 values for <data_type>. The most likely use consists in dynamically entering
1632 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1633 IP address or affect its quality of service. It is possible to pass multiple
1634 data_types in a single call.
1635
1636set timeout cli <delay>
1637 Change the CLI interface timeout for current connection. This can be useful
1638 during long debugging sessions where the user needs to constantly inspect
1639 some indicators without being disconnected. The delay is passed in seconds.
1640
1641set weight <backend>/<server> <weight>[%]
1642 Change a server's weight to the value passed in argument. If the value ends
1643 with the '%' sign, then the new weight will be relative to the initially
1644 configured weight. Absolute weights are permitted between 0 and 256.
1645 Relative weights must be positive with the resulting absolute weight is
1646 capped at 256. Servers which are part of a farm running a static
1647 load-balancing algorithm have stricter limitations because the weight
1648 cannot change once set. Thus for these servers, the only accepted values
1649 are 0 and 100% (or 0 and the initial weight). Changes take effect
1650 immediately, though certain LB algorithms require a certain amount of
1651 requests to consider changes. A typical usage of this command is to
1652 disable a server during an update by setting its weight to zero, then to
1653 enable it again after the update by setting it back to 100%. This command
1654 is restricted and can only be issued on sockets configured for level
1655 "admin". Both the backend and the server may be specified either by their
1656 name or by their numeric ID, prefixed with a sharp ('#').
1657
Willy Tarreauae795722016-02-16 11:27:28 +01001658show env [<name>]
1659 Dump one or all environment variables known by the process. Without any
1660 argument, all variables are dumped. With an argument, only the specified
1661 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1662 Variables are dumped in the same format as they are stored or returned by the
1663 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1664 certain configuration files making heavy use of environment variables to
1665 ensure that they contain the expected values. This command is restricted and
1666 can only be issued on sockets configured for levels "operator" or "admin".
1667
Willy Tarreau44aed902015-10-13 14:45:29 +02001668show errors [<iid>]
1669 Dump last known request and response errors collected by frontends and
1670 backends. If <iid> is specified, the limit the dump to errors concerning
1671 either frontend or backend whose ID is <iid>. This command is restricted
1672 and can only be issued on sockets configured for levels "operator" or
1673 "admin".
1674
1675 The errors which may be collected are the last request and response errors
1676 caused by protocol violations, often due to invalid characters in header
1677 names. The report precisely indicates what exact character violated the
1678 protocol. Other important information such as the exact date the error was
1679 detected, frontend and backend names, the server name (when known), the
1680 internal session ID and the source address which has initiated the session
1681 are reported too.
1682
1683 All characters are returned, and non-printable characters are encoded. The
1684 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1685 letter following a backslash. The backslash itself is encoded as '\\' to
1686 avoid confusion. Other non-printable characters are encoded '\xNN' where
1687 NN is the two-digits hexadecimal representation of the character's ASCII
1688 code.
1689
1690 Lines are prefixed with the position of their first character, starting at 0
1691 for the beginning of the buffer. At most one input line is printed per line,
1692 and large lines will be broken into multiple consecutive output lines so that
1693 the output never goes beyond 79 characters wide. It is easy to detect if a
1694 line was broken, because it will not end with '\n' and the next line's offset
1695 will be followed by a '+' sign, indicating it is a continuation of previous
1696 line.
1697
1698 Example :
1699 $ echo "show errors" | socat stdio /tmp/sock1
1700 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1701 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1702 response length 213 bytes, error at position 23:
1703
1704 00000 HTTP/1.0 200 OK\r\n
1705 00017 header/bizarre:blah\r\n
1706 00038 Location: blah\r\n
1707 00054 Long-line: this is a very long line which should b
1708 00104+ e broken into multiple lines on the output buffer,
1709 00154+ otherwise it would be too large to print in a ter
1710 00204+ minal\r\n
1711 00211 \r\n
1712
1713 In the example above, we see that the backend "http-in" which has internal
1714 ID 2 has blocked an invalid response from its server s2 which has internal
1715 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1716 received by frontend fe-eth0 whose ID is 1. The total response length was
1717 213 bytes when the error was detected, and the error was at byte 23. This
1718 is the slash ('/') in header name "header/bizarre", which is not a valid
1719 HTTP character for a header name.
1720
1721show backend
1722 Dump the list of backends available in the running process
1723
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001724show info [typed]
1725 Dump info about haproxy status on current process. If "typed" is passed as an
1726 optional argument, field numbers, names and types are emitted as well so that
1727 external monitoring products can easily retrieve, possibly aggregate, then
1728 report information found in fields they don't know. Each field is dumped on
1729 its own line. By default, the format contains only two columns delimited by a
1730 colon (':'). The left one is the field name and the right one is the value.
1731 It is very important to note that in typed output format, the dump for a
1732 single object is contigous so that there is no need for a consumer to store
1733 everything at once.
1734
1735 When using the typed output format, each line is made of 4 columns delimited
1736 by colons (':'). The first column is a dot-delimited series of 3 elements. The
1737 first element is the numeric position of the field in the list (starting at
1738 zero). This position shall not change over time, but holes are to be expected,
1739 depending on build options or if some fields are deleted in the future. The
1740 second element is the field name as it appears in the default "show info"
1741 output. The third element is the relative process number starting at 1.
1742
1743 The rest of the line starting after the first colon follows the "typed output
1744 format" described in the section above. In short, the second column (after the
1745 first ':') indicates the origin, nature and scope of the variable. The third
1746 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1747 "str". Then the fourth column is the value itself, which the consumer knows
1748 how to parse thanks to column 3 and how to process thanks to column 2.
1749
1750 Thus the overall line format in typed mode is :
1751
1752 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
1753
1754 Example :
1755
1756 > show info
1757 Name: HAProxy
1758 Version: 1.7-dev1-de52ea-146
1759 Release_date: 2016/03/11
1760 Nbproc: 1
1761 Process_num: 1
1762 Pid: 28105
1763 Uptime: 0d 0h00m04s
1764 Uptime_sec: 4
1765 Memmax_MB: 0
1766 PoolAlloc_MB: 0
1767 PoolUsed_MB: 0
1768 PoolFailed: 0
1769 (...)
1770
1771 > show info typed
1772 0.Name.1:POS:str:HAProxy
1773 1.Version.1:POS:str:1.7-dev1-de52ea-146
1774 2.Release_date.1:POS:str:2016/03/11
1775 3.Nbproc.1:CGS:u32:1
1776 4.Process_num.1:KGP:u32:1
1777 5.Pid.1:SGP:u32:28105
1778 6.Uptime.1:MDP:str:0d 0h00m08s
1779 7.Uptime_sec.1:MDP:u32:8
1780 8.Memmax_MB.1:CLP:u32:0
1781 9.PoolAlloc_MB.1:MGP:u32:0
1782 10.PoolUsed_MB.1:MGP:u32:0
1783 11.PoolFailed.1:MCP:u32:0
1784 (...)
1785
1786 In the typed format, the presence of the process ID at the end of the line
1787 makes it very easy to visually aggregate outputs from multiple processes.
1788 Example :
1789
1790 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
1791 echo show info typed | socat /var/run/haproxy.sock2 ) | \
1792 sort -t . -k 1,1n -k 2,2 -k 3,3n
1793 0.Name.1:POS:str:HAProxy
1794 0.Name.2:POS:str:HAProxy
1795 1.Version.1:POS:str:1.7-dev1-868ab3-148
1796 1.Version.2:POS:str:1.7-dev1-868ab3-148
1797 2.Release_date.1:POS:str:2016/03/11
1798 2.Release_date.2:POS:str:2016/03/11
1799 3.Nbproc.1:CGS:u32:2
1800 3.Nbproc.2:CGS:u32:2
1801 4.Process_num.1:KGP:u32:1
1802 4.Process_num.2:KGP:u32:2
1803 5.Pid.1:SGP:u32:30120
1804 5.Pid.2:SGP:u32:30121
1805 6.Uptime.1:MDP:str:0d 0h01m28s
1806 6.Uptime.2:MDP:str:0d 0h01m28s
1807 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02001808
1809show map [<map>]
1810 Dump info about map converters. Without argument, the list of all available
1811 maps is returned. If a <map> is specified, its contents are dumped. <map> is
1812 the #<id> or <file>. The first column is a unique identifier. It can be used
1813 as reference for the operation "del map" and "set map". The second column is
1814 the pattern and the third column is the sample if available. The data returned
1815 are not directly a list of available maps, but are the list of all patterns
1816 composing any map. Many of these patterns can be shared with ACL.
1817
1818show acl [<acl>]
1819 Dump info about acl converters. Without argument, the list of all available
1820 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1821 the #<id> or <file>. The dump format is the same than the map even for the
1822 sample value. The data returned are not a list of available ACL, but are the
1823 list of all patterns composing any ACL. Many of these patterns can be shared
1824 with maps.
1825
1826show pools
1827 Dump the status of internal memory pools. This is useful to track memory
1828 usage when suspecting a memory leak for example. It does exactly the same
1829 as the SIGQUIT when running in foreground except that it does not flush
1830 the pools.
1831
1832show servers state [<backend>]
1833 Dump the state of the servers found in the running configuration. A backend
1834 name or identifier may be provided to limit the output to this backend only.
1835
1836 The dump has the following format:
1837 - first line contains the format version (1 in this specification);
1838 - second line contains the column headers, prefixed by a sharp ('#');
1839 - third line and next ones contain data;
1840 - each line starting by a sharp ('#') is considered as a comment.
1841
1842 Since multiple versions of the ouptput may co-exist, below is the list of
1843 fields and their order per file format version :
1844 1:
1845 be_id: Backend unique id.
1846 be_name: Backend label.
1847 srv_id: Server unique id (in the backend).
1848 srv_name: Server label.
1849 srv_addr: Server IP address.
1850 srv_op_state: Server operational state (UP/DOWN/...).
1851 In source code: SRV_ST_*.
1852 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
1853 In source code: SRV_ADMF_*.
1854 srv_uweight: User visible server's weight.
1855 srv_iweight: Server's initial weight.
1856 srv_time_since_last_change: Time since last operational change.
1857 srv_check_status: Last health check status.
1858 srv_check_result: Last check result (FAILED/PASSED/...).
1859 In source code: CHK_RES_*.
1860 srv_check_health: Checks rise / fall current counter.
1861 srv_check_state: State of the check (ENABLED/PAUSED/...).
1862 In source code: CHK_ST_*.
1863 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
1864 In source code: CHK_ST_*.
1865 bk_f_forced_id: Flag to know if the backend ID is forced by
1866 configuration.
1867 srv_f_forced_id: Flag to know if the server's ID is forced by
1868 configuration.
1869
1870show sess
1871 Dump all known sessions. Avoid doing this on slow connections as this can
1872 be huge. This command is restricted and can only be issued on sockets
1873 configured for levels "operator" or "admin".
1874
1875show sess <id>
1876 Display a lot of internal information about the specified session identifier.
1877 This identifier is the first field at the beginning of the lines in the dumps
1878 of "show sess" (it corresponds to the session pointer). Those information are
1879 useless to most users but may be used by haproxy developers to troubleshoot a
1880 complex bug. The output format is intentionally not documented so that it can
1881 freely evolve depending on demands. You may find a description of all fields
1882 returned in src/dumpstats.c
1883
1884 The special id "all" dumps the states of all sessions, which must be avoided
1885 as much as possible as it is highly CPU intensive and can take a lot of time.
1886
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001887show stat [<iid> <type> <sid>] [typed]
1888 Dump statistics using the CSV format, or using the extended typed output
1889 format described in the section above if "typed" is passed after the other
1890 arguments. By passing <id>, <type> and <sid>, it is possible to dump only
1891 selected items :
Willy Tarreau44aed902015-10-13 14:45:29 +02001892 - <iid> is a proxy ID, -1 to dump everything
1893 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
1894 backends, 4 for servers, -1 for everything. These values can be ORed,
1895 for example:
1896 1 + 2 = 3 -> frontend + backend.
1897 1 + 2 + 4 = 7 -> frontend + backend + server.
1898 - <sid> is a server ID, -1 to dump everything from the selected proxy.
1899
1900 Example :
1901 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
1902 >>> Name: HAProxy
1903 Version: 1.4-dev2-49
1904 Release_date: 2009/09/23
1905 Nbproc: 1
1906 Process_num: 1
1907 (...)
1908
1909 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
1910 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
1911 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
1912 (...)
1913 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
1914
1915 $
1916
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001917 In this example, two commands have been issued at once. That way it's easy to
1918 find which process the stats apply to in multi-process mode. This is not
1919 needed in the typed output format as the process number is reported on each
1920 line. Notice the empty line after the information output which marks the end
1921 of the first block. A similar empty line appears at the end of the second
1922 block (stats) so that the reader knows the output has not been truncated.
1923
1924 When "typed" is specified, the output format is more suitable to monitoring
1925 tools because it provides numeric positions and indicates the type of each
1926 output field. Each value stands on its own line with process number, element
1927 number, nature, origin and scope. This same format is available via the HTTP
1928 stats by passing ";typed" after the URI. It is very important to note that in
1929 typed output format, the dump for a single object is contigous so that there
1930 is no need for a consumer to store everything at once.
1931
1932 When using the typed output format, each line is made of 4 columns delimited
1933 by colons (':'). The first column is a dot-delimited series of 5 elements. The
1934 first element is a letter indicating the type of the object being described.
1935 At the moment the following object types are known : 'F' for a frontend, 'B'
1936 for a backend, 'L' for a listener, and 'S' for a server. The second element
1937 The second element is a positive integer representing the unique identifier of
1938 the proxy the object belongs to. It is equivalent to the "iid" column of the
1939 CSV output and matches the value in front of the optional "id" directive found
1940 in the frontend or backend section. The third element is a positive integer
1941 containing the unique object identifier inside the proxy, and corresponds to
1942 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
1943 or a backend. For a listener or a server, this corresponds to their respective
1944 ID inside the proxy. The fourth element is the numeric position of the field
1945 in the list (starting at zero). This position shall not change over time, but
1946 holes are to be expected, depending on build options or if some fields are
1947 deleted in the future. The fifth element is the field name as it appears in
1948 the CSV output. The sixth element is a positive integer and is the relative
1949 process number starting at 1.
1950
1951 The rest of the line starting after the first colon follows the "typed output
1952 format" described in the section above. In short, the second column (after the
1953 first ':') indicates the origin, nature and scope of the variable. The third
1954 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1955 "str". Then the fourth column is the value itself, which the consumer knows
1956 how to parse thanks to column 3 and how to process thanks to column 2.
1957
1958 Thus the overall line format in typed mode is :
1959
1960 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
1961
1962 Here's an example of typed output format :
1963
1964 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
1965 F.2.0.0.pxname.1:MGP:str:private-frontend
1966 F.2.0.1.svname.1:MGP:str:FRONTEND
1967 F.2.0.8.bin.1:MGP:u64:0
1968 F.2.0.9.bout.1:MGP:u64:0
1969 F.2.0.40.hrsp_2xx.1:MGP:u64:0
1970 L.2.1.0.pxname.1:MGP:str:private-frontend
1971 L.2.1.1.svname.1:MGP:str:sock-1
1972 L.2.1.17.status.1:MGP:str:OPEN
1973 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
1974 S.3.13.60.rtime.1:MCP:u32:0
1975 S.3.13.61.ttime.1:MCP:u32:0
1976 S.3.13.62.agent_status.1:MGP:str:L4TOUT
1977 S.3.13.64.agent_duration.1:MGP:u64:2001
1978 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
1979 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
1980 S.3.13.67.check_rise.1:MCP:u32:2
1981 S.3.13.68.check_fall.1:MCP:u32:3
1982 S.3.13.69.check_health.1:SGP:u32:0
1983 S.3.13.70.agent_rise.1:MaP:u32:1
1984 S.3.13.71.agent_fall.1:SGP:u32:1
1985 S.3.13.72.agent_health.1:SGP:u32:1
1986 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
1987 S.3.13.75.mode.1:MAP:str:http
1988 B.3.0.0.pxname.1:MGP:str:private-backend
1989 B.3.0.1.svname.1:MGP:str:BACKEND
1990 B.3.0.2.qcur.1:MGP:u32:0
1991 B.3.0.3.qmax.1:MGP:u32:0
1992 B.3.0.4.scur.1:MGP:u32:0
1993 B.3.0.5.smax.1:MGP:u32:0
1994 B.3.0.6.slim.1:MGP:u32:1000
1995 B.3.0.55.lastsess.1:MMP:s32:-1
1996 (...)
1997
1998 In the typed format, the presence of the process ID at the end of the line
1999 makes it very easy to visually aggregate outputs from multiple processes, as
2000 show in the example below where each line appears for each process :
2001
2002 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2003 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2004 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2005 B.3.0.0.pxname.1:MGP:str:private-backend
2006 B.3.0.0.pxname.2:MGP:str:private-backend
2007 B.3.0.1.svname.1:MGP:str:BACKEND
2008 B.3.0.1.svname.2:MGP:str:BACKEND
2009 B.3.0.2.qcur.1:MGP:u32:0
2010 B.3.0.2.qcur.2:MGP:u32:0
2011 B.3.0.3.qmax.1:MGP:u32:0
2012 B.3.0.3.qmax.2:MGP:u32:0
2013 B.3.0.4.scur.1:MGP:u32:0
2014 B.3.0.4.scur.2:MGP:u32:0
2015 B.3.0.5.smax.1:MGP:u32:0
2016 B.3.0.5.smax.2:MGP:u32:0
2017 B.3.0.6.slim.1:MGP:u32:1000
2018 B.3.0.6.slim.2:MGP:u32:1000
2019 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002020
2021show stat resolvers [<resolvers section id>]
2022 Dump statistics for the given resolvers section, or all resolvers sections
2023 if no section is supplied.
2024
2025 For each name server, the following counters are reported:
2026 sent: number of DNS requests sent to this server
2027 valid: number of DNS valid responses received from this server
2028 update: number of DNS responses used to update the server's IP address
2029 cname: number of CNAME responses
2030 cname_error: CNAME errors encountered with this server
2031 any_err: number of empty response (IE: server does not support ANY type)
2032 nx: non existent domain response received from this server
2033 timeout: how many time this server did not answer in time
2034 refused: number of requests refused by this server
2035 other: any other DNS errors
2036 invalid: invalid DNS response (from a protocol point of view)
2037 too_big: too big response
2038 outdated: number of response arrived too late (after an other name server)
2039
2040show table
2041 Dump general information on all known stick-tables. Their name is returned
2042 (the name of the proxy which holds them), their type (currently zero, always
2043 IP), their size in maximum possible number of entries, and the number of
2044 entries currently in use.
2045
2046 Example :
2047 $ echo "show table" | socat stdio /tmp/sock1
2048 >>> # table: front_pub, type: ip, size:204800, used:171454
2049 >>> # table: back_rdp, type: ip, size:204800, used:0
2050
2051show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2052 Dump contents of stick-table <name>. In this mode, a first line of generic
2053 information about the table is reported as with "show table", then all
2054 entries are dumped. Since this can be quite heavy, it is possible to specify
2055 a filter in order to specify what entries to display.
2056
2057 When the "data." form is used the filter applies to the stored data (see
2058 "stick-table" in section 4.2). A stored data type must be specified
2059 in <type>, and this data type must be stored in the table otherwise an
2060 error is reported. The data is compared according to <operator> with the
2061 64-bit integer <value>. Operators are the same as with the ACLs :
2062
2063 - eq : match entries whose data is equal to this value
2064 - ne : match entries whose data is not equal to this value
2065 - le : match entries whose data is less than or equal to this value
2066 - ge : match entries whose data is greater than or equal to this value
2067 - lt : match entries whose data is less than this value
2068 - gt : match entries whose data is greater than this value
2069
2070
2071 When the key form is used the entry <key> is shown. The key must be of the
2072 same type as the table, which currently is limited to IPv4, IPv6, integer,
2073 and string.
2074
2075 Example :
2076 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2077 >>> # table: http_proxy, type: ip, size:204800, used:2
2078 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2079 bytes_out_rate(60000)=187
2080 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2081 bytes_out_rate(60000)=191
2082
2083 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2084 >>> # table: http_proxy, type: ip, size:204800, used:2
2085 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2086 bytes_out_rate(60000)=191
2087
2088 $ echo "show table http_proxy data.conn_rate gt 5" | \
2089 socat stdio /tmp/sock1
2090 >>> # table: http_proxy, type: ip, size:204800, used:2
2091 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2092 bytes_out_rate(60000)=191
2093
2094 $ echo "show table http_proxy key 127.0.0.2" | \
2095 socat stdio /tmp/sock1
2096 >>> # table: http_proxy, type: ip, size:204800, used:2
2097 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2098 bytes_out_rate(60000)=191
2099
2100 When the data criterion applies to a dynamic value dependent on time such as
2101 a bytes rate, the value is dynamically computed during the evaluation of the
2102 entry in order to decide whether it has to be dumped or not. This means that
2103 such a filter could match for some time then not match anymore because as
2104 time goes, the average event rate drops.
2105
2106 It is possible to use this to extract lists of IP addresses abusing the
2107 service, in order to monitor them or even blacklist them in a firewall.
2108 Example :
2109 $ echo "show table http_proxy data.gpc0 gt 0" \
2110 | socat stdio /tmp/sock1 \
2111 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2112 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2113
2114show tls-keys
2115 Dump all loaded TLS ticket keys. The TLS ticket key reference ID and the
2116 file from which the keys have been loaded is shown. Both of those can be
2117 used to update the TLS keys using "set ssl tls-key".
2118
2119shutdown frontend <frontend>
2120 Completely delete the specified frontend. All the ports it was bound to will
2121 be released. It will not be possible to enable the frontend anymore after
2122 this operation. This is intended to be used in environments where stopping a
2123 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2124 way it's possible to release the port and bind it into another process to
2125 restore operations. The frontend will not appear at all on the stats page
2126 once it is terminated.
2127
2128 The frontend may be specified either by its name or by its numeric ID,
2129 prefixed with a sharp ('#').
2130
2131 This command is restricted and can only be issued on sockets configured for
2132 level "admin".
2133
2134shutdown session <id>
2135 Immediately terminate the session matching the specified session identifier.
2136 This identifier is the first field at the beginning of the lines in the dumps
2137 of "show sess" (it corresponds to the session pointer). This can be used to
2138 terminate a long-running session without waiting for a timeout or when an
2139 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2140 flag in the logs.
2141
2142shutdown sessions server <backend>/<server>
2143 Immediately terminate all the sessions attached to the specified server. This
2144 can be used to terminate long-running sessions after a server is put into
2145 maintenance mode, for instance. Such terminated sessions are reported with a
2146 'K' flag in the logs.
2147
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002148
214910. Tricks for easier configuration management
2150----------------------------------------------
2151
2152It is very common that two HAProxy nodes constituting a cluster share exactly
2153the same configuration modulo a few addresses. Instead of having to maintain a
2154duplicate configuration for each node, which will inevitably diverge, it is
2155possible to include environment variables in the configuration. Thus multiple
2156configuration may share the exact same file with only a few different system
2157wide environment variables. This started in version 1.5 where only addresses
2158were allowed to include environment variables, and 1.6 goes further by
2159supporting environment variables everywhere. The syntax is the same as in the
2160UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2161curly brace ('{'), then the variable name followed by the closing brace ('}').
2162Except for addresses, environment variables are only interpreted in arguments
2163surrounded with double quotes (this was necessary not to break existing setups
2164using regular expressions involving the dollar symbol).
2165
2166Environment variables also make it convenient to write configurations which are
2167expected to work on various sites where only the address changes. It can also
2168permit to remove passwords from some configs. Example below where the the file
2169"site1.env" file is sourced by the init script upon startup :
2170
2171 $ cat site1.env
2172 LISTEN=192.168.1.1
2173 CACHE_PFX=192.168.11
2174 SERVER_PFX=192.168.22
2175 LOGGER=192.168.33.1
2176 STATSLP=admin:pa$$w0rd
2177 ABUSERS=/etc/haproxy/abuse.lst
2178 TIMEOUT=10s
2179
2180 $ cat haproxy.cfg
2181 global
2182 log "${LOGGER}:514" local0
2183
2184 defaults
2185 mode http
2186 timeout client "${TIMEOUT}"
2187 timeout server "${TIMEOUT}"
2188 timeout connect 5s
2189
2190 frontend public
2191 bind "${LISTEN}:80"
2192 http-request reject if { src -f "${ABUSERS}" }
2193 stats uri /stats
2194 stats auth "${STATSLP}"
2195 use_backend cache if { path_end .jpg .css .ico }
2196 default_backend server
2197
2198 backend cache
2199 server cache1 "${CACHE_PFX}.1:18080" check
2200 server cache2 "${CACHE_PFX}.2:18080" check
2201
2202 backend server
2203 server cache1 "${SERVER_PFX}.1:8080" check
2204 server cache2 "${SERVER_PFX}.2:8080" check
2205
2206
220711. Well-known traps to avoid
2208-----------------------------
2209
2210Once in a while, someone reports that after a system reboot, the haproxy
2211service wasn't started, and that once they start it by hand it works. Most
2212often, these people are running a clustered IP address mechanism such as
2213keepalived, to assign the service IP address to the master node only, and while
2214it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2215working after they bound it to the virtual IP address. What happens here is
2216that when the service starts, the virtual IP address is not yet owned by the
2217local node, so when HAProxy wants to bind to it, the system rejects this
2218because it is not a local IP address. The fix doesn't consist in delaying the
2219haproxy service startup (since it wouldn't stand a restart), but instead to
2220properly configure the system to allow binding to non-local addresses. This is
2221easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2222is also needed in order to transparently intercept the IP traffic that passes
2223through HAProxy for a specific target address.
2224
2225Multi-process configurations involving source port ranges may apparently seem
2226to work but they will cause some random failures under high loads because more
2227than one process may try to use the same source port to connect to the same
2228server, which is not possible. The system will report an error and a retry will
2229happen, picking another port. A high value in the "retries" parameter may hide
2230the effect to a certain extent but this also comes with increased CPU usage and
2231processing time. Logs will also report a certain number of retries. For this
2232reason, port ranges should be avoided in multi-process configurations.
2233
2234Since HAProxy uses SO_REUSEPORT and supports having multiple independant
2235processes bound to the same IP:port, during troubleshooting it can happen that
2236an old process was not stopped before a new one was started. This provides
2237absurd test results which tend to indicate that any change to the configuration
2238is ignored. The reason is that in fact even the new process is restarted with a
2239new configuration, the old one also gets some incoming connections and
2240processes them, returning unexpected results. When in doubt, just stop the new
2241process and try again. If it still works, it very likely means that an old
2242process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2243help here.
2244
2245When adding entries to an ACL from the command line (eg: when blacklisting a
2246source address), it is important to keep in mind that these entries are not
2247synchronized to the file and that if someone reloads the configuration, these
2248updates will be lost. While this is often the desired effect (for blacklisting)
2249it may not necessarily match expectations when the change was made as a fix for
2250a problem. See the "add acl" action of the CLI interface.
2251
2252
225312. Debugging and performance issues
2254------------------------------------
2255
2256When HAProxy is started with the "-d" option, it will stay in the foreground
2257and will print one line per event, such as an incoming connection, the end of a
2258connection, and for each request or response header line seen. This debug
2259output is emitted before the contents are processed, so they don't consider the
2260local modifications. The main use is to show the request and response without
2261having to run a network sniffer. The output is less readable when multiple
2262connections are handled in parallel, though the "debug2ansi" and "debug2html"
2263scripts found in the examples/ directory definitely help here by coloring the
2264output.
2265
2266If a request or response is rejected because HAProxy finds it is malformed, the
2267best thing to do is to connect to the CLI and issue "show errors", which will
2268report the last captured faulty request and response for each frontend and
2269backend, with all the necessary information to indicate precisely the first
2270character of the input stream that was rejected. This is sometimes needed to
2271prove to customers or to developers that a bug is present in their code. In
2272this case it is often possible to relax the checks (but still keep the
2273captures) using "option accept-invalid-http-request" or its equivalent for
2274responses coming from the server "option accept-invalid-http-response". Please
2275see the configuration manual for more details.
2276
2277Example :
2278
2279 > show errors
2280 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2281
2282 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2283 backend <NONE> (#-1), server <NONE> (#-1), event #0
2284 src 127.0.0.1:51981, session #0, session flags 0x00000080
2285 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2286 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2287 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2288 pending 31 bytes, wrapping at 8040, error at position 13:
2289
2290 00000 GET /invalid request HTTP/1.1\r\n
2291
2292
2293The output of "show info" on the CLI provides a number of useful information
2294regarding the maximum connection rate ever reached, maximum SSL key rate ever
2295reached, and in general all information which can help to explain temporary
2296issues regarding CPU or memory usage. Example :
2297
2298 > show info
2299 Name: HAProxy
2300 Version: 1.6-dev7-e32d18-17
2301 Release_date: 2015/10/12
2302 Nbproc: 1
2303 Process_num: 1
2304 Pid: 7949
2305 Uptime: 0d 0h02m39s
2306 Uptime_sec: 159
2307 Memmax_MB: 0
2308 Ulimit-n: 120032
2309 Maxsock: 120032
2310 Maxconn: 60000
2311 Hard_maxconn: 60000
2312 CurrConns: 0
2313 CumConns: 3
2314 CumReq: 3
2315 MaxSslConns: 0
2316 CurrSslConns: 0
2317 CumSslConns: 0
2318 Maxpipes: 0
2319 PipesUsed: 0
2320 PipesFree: 0
2321 ConnRate: 0
2322 ConnRateLimit: 0
2323 MaxConnRate: 1
2324 SessRate: 0
2325 SessRateLimit: 0
2326 MaxSessRate: 1
2327 SslRate: 0
2328 SslRateLimit: 0
2329 MaxSslRate: 0
2330 SslFrontendKeyRate: 0
2331 SslFrontendMaxKeyRate: 0
2332 SslFrontendSessionReuse_pct: 0
2333 SslBackendKeyRate: 0
2334 SslBackendMaxKeyRate: 0
2335 SslCacheLookups: 0
2336 SslCacheMisses: 0
2337 CompressBpsIn: 0
2338 CompressBpsOut: 0
2339 CompressBpsRateLim: 0
2340 ZlibMemUsage: 0
2341 MaxZlibMemUsage: 0
2342 Tasks: 5
2343 Run_queue: 1
2344 Idle_pct: 100
2345 node: wtap
2346 description:
2347
2348When an issue seems to randomly appear on a new version of HAProxy (eg: every
2349second request is aborted, occasional crash, etc), it is worth trying to enable
2350memory poisonning so that each call to malloc() is immediately followed by the
2351filling of the memory area with a configurable byte. By default this byte is
23520x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2353will have the same effect as a calloc() and which may make issues disappear).
2354Memory poisonning is enabled on the command line using the "-dM" option. It
2355slightly hurts performance and is not recommended for use in production. If
2356an issue happens all the time with it or never happens when poisoonning uses
2357byte zero, it clearly means you've found a bug and you definitely need to
2358report it. Otherwise if there's no clear change, the problem it is not related.
2359
2360When debugging some latency issues, it is important to use both strace and
2361tcpdump on the local machine, and another tcpdump on the remote system. The
2362reason for this is that there are delays everywhere in the processing chain and
2363it is important to know which one is causing latency to know where to act. In
2364practice, the local tcpdump will indicate when the input data come in. Strace
2365will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2366openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2367show when haproxy sends the data, and tcpdump will show when the system sends
2368these data to the interface. Then the external tcpdump will show when the data
2369sent are really received (since the local one only shows when the packets are
2370queued). The benefit of sniffing on the local system is that strace and tcpdump
2371will use the same reference clock. Strace should be used with "-tts200" to get
2372complete timestamps and report large enough chunks of data to read them.
2373Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2374numbers and complete timestamps.
2375
2376In practice, received data are almost always immediately received by haproxy
2377(unless the machine has a saturated CPU or these data are invalid and not
2378delivered). If these data are received but not sent, it generally is because
2379the output buffer is saturated (ie: recipient doesn't consume the data fast
2380enough). This can be confirmed by seeing that the polling doesn't notify of
2381the ability to write on the output file descriptor for some time (it's often
2382easier to spot in the strace output when the data finally leave and then roll
2383back to see when the write event was notified). It generally matches an ACK
2384received from the recipient, and detected by tcpdump. Once the data are sent,
2385they may spend some time in the system doing nothing. Here again, the TCP
2386congestion window may be limited and not allow these data to leave, waiting for
2387an ACK to open the window. If the traffic is idle and the data take 40 ms or
2388200 ms to leave, it's a different issue (which is not an issue), it's the fact
2389that the Nagle algorithm prevents empty packets from leaving immediately, in
2390hope that they will be merged with subsequent data. HAProxy automatically
2391disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2392enabled when forwarding an HTTP body (and this contributes to the performance
2393improvement there by reducing the number of packets). Some HTTP non-compliant
2394applications may be sensitive to the latency when delivering incomplete HTTP
2395response messages. In this case you will have to enable "option http-no-delay"
2396to disable Nagle in order to work around their design, keeping in mind that any
2397other proxy in the chain may similarly be impacted. If tcpdump reports that data
2398leave immediately but the other end doesn't see them quickly, it can mean there
2399is a congestionned WAN link, a congestionned LAN with flow control enabled and
2400preventing the data from leaving, or more commonly that HAProxy is in fact
2401running in a virtual machine and that for whatever reason the hypervisor has
2402decided that the data didn't need to be sent immediately. In virtualized
2403environments, latency issues are almost always caused by the virtualization
2404layer, so in order to save time, it's worth first comparing tcpdump in the VM
2405and on the external components. Any difference has to be credited to the
2406hypervisor and its accompanying drivers.
2407
2408When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2409means that the side sending them has got the proof of a lost packet. While not
2410seeing them doesn't mean there are no losses, seeing them definitely means the
2411network is lossy. Losses are normal on a network, but at a rate where SACKs are
2412not noticeable at the naked eye. If they appear a lot in the traces, it is
2413worth investigating exactly what happens and where the packets are lost. HTTP
2414doesn't cope well with TCP losses, which introduce huge latencies.
2415
2416The "netstat -i" command will report statistics per interface. An interface
2417where the Rx-Ovr counter grows indicates that the system doesn't have enough
2418resources to receive all incoming packets and that they're lost before being
2419processed by the network driver. Rx-Drp indicates that some received packets
2420were lost in the network stack because the application doesn't process them
2421fast enough. This can happen during some attacks as well. Tx-Drp means that
2422the output queues were full and packets had to be dropped. When using TCP it
2423should be very rare, but will possibly indicte a saturated outgoing link.
2424
2425
242613. Security considerations
2427---------------------------
2428
2429HAProxy is designed to run with very limited privileges. The standard way to
2430use it is to isolate it into a chroot jail and to drop its privileges to a
2431non-root user without any permissions inside this jail so that if any future
2432vulnerability were to be discovered, its compromise would not affect the rest
2433of the system.
2434
2435In order to perfom a chroot, it first needs to be started as a root user. It is
2436pointless to build hand-made chroots to start the process there, these ones are
2437painful to build, are never properly maintained and always contain way more
2438bugs than the main file-system. And in case of compromise, the intruder can use
2439the purposely built file-system. Unfortunately many administrators confuse
2440"start as root" and "run as root", resulting in the uid change to be done prior
2441to starting haproxy, and reducing the effective security restrictions.
2442
2443HAProxy will need to be started as root in order to :
2444 - adjust the file descriptor limits
2445 - bind to privileged port numbers
2446 - bind to a specific network interface
2447 - transparently listen to a foreign address
2448 - isolate itself inside the chroot jail
2449 - drop to another non-privileged UID
2450
2451HAProxy may require to be run as root in order to :
2452 - bind to an interface for outgoing connections
2453 - bind to privileged source ports for outgoing connections
2454 - transparently bind to a foreing address for outgoing connections
2455
2456Most users will never need the "run as root" case. But the "start as root"
2457covers most usages.
2458
2459A safe configuration will have :
2460
2461 - a chroot statement pointing to an empty location without any access
2462 permissions. This can be prepared this way on the UNIX command line :
2463
2464 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
2465
2466 and referenced like this in the HAProxy configuration's global section :
2467
2468 chroot /var/empty
2469
2470 - both a uid/user and gid/group statements in the global section :
2471
2472 user haproxy
2473 group haproxy
2474
2475 - a stats socket whose mode, uid and gid are set to match the user and/or
2476 group allowed to access the CLI so that nobody may access it :
2477
2478 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
2479