Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 1 | ------------------------ |
| 2 | HAProxy Management Guide |
| 3 | ------------------------ |
Willy Tarreau | 8234f6d | 2016-03-14 00:10:05 +0100 | [diff] [blame] | 4 | version 1.7 |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 5 | |
| 6 | |
| 7 | This document describes how to start, stop, manage, and troubleshoot HAProxy, |
| 8 | as well as some known limitations and traps to avoid. It does not describe how |
| 9 | to configure it (for this please read configuration.txt). |
| 10 | |
| 11 | Note to documentation contributors : |
| 12 | This document is formatted with 80 columns per line, with even number of |
| 13 | spaces for indentation and without tabs. Please follow these rules strictly |
| 14 | so that it remains easily printable everywhere. If you add sections, please |
| 15 | update the summary below for easier searching. |
| 16 | |
| 17 | |
| 18 | Summary |
| 19 | ------- |
| 20 | |
| 21 | 1. Prerequisites |
| 22 | 2. Quick reminder about HAProxy's architecture |
| 23 | 3. Starting HAProxy |
| 24 | 4. Stopping and restarting HAProxy |
| 25 | 5. File-descriptor limitations |
| 26 | 6. Memory management |
| 27 | 7. CPU usage |
| 28 | 8. Logging |
| 29 | 9. Statistics and monitoring |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 30 | 9.1. CSV format |
Willy Tarreau | 5d8b979 | 2016-03-11 11:09:34 +0100 | [diff] [blame] | 31 | 9.2. Typed output format |
| 32 | 9.3. Unix Socket commands |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 33 | 10. Tricks for easier configuration management |
| 34 | 11. Well-known traps to avoid |
| 35 | 12. Debugging and performance issues |
| 36 | 13. Security considerations |
| 37 | |
| 38 | |
| 39 | 1. Prerequisites |
| 40 | ---------------- |
| 41 | |
| 42 | In this document it is assumed that the reader has sufficient administration |
| 43 | skills on a UNIX-like operating system, uses the shell on a daily basis and is |
| 44 | familiar with troubleshooting utilities such as strace and tcpdump. |
| 45 | |
| 46 | |
| 47 | 2. Quick reminder about HAProxy's architecture |
| 48 | ---------------------------------------------- |
| 49 | |
| 50 | HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is |
| 51 | uses event multiplexing to schedule all of its activities instead of relying on |
| 52 | the system to schedule between multiple activities. Most of the time it runs as |
| 53 | a single process, so the output of "ps aux" on a system will report only one |
| 54 | "haproxy" process, unless a soft reload is in progress and an older process is |
| 55 | finishing its job in parallel to the new one. It is thus always easy to trace |
| 56 | its activity using the strace utility. |
| 57 | |
| 58 | HAProxy is designed to isolate itself into a chroot jail during startup, where |
| 59 | it cannot perform any file-system access at all. This is also true for the |
| 60 | libraries it depends on (eg: libc, libssl, etc). The immediate effect is that |
| 61 | a running process will not be able to reload a configuration file to apply |
| 62 | changes, instead a new process will be started using the updated configuration |
| 63 | file. Some other less obvious effects are that some timezone files or resolver |
| 64 | files the libc might attempt to access at run time will not be found, though |
| 65 | this should generally not happen as they're not needed after startup. A nice |
| 66 | consequence of this principle is that the HAProxy process is totally stateless, |
| 67 | and no cleanup is needed after it's killed, so any killing method that works |
| 68 | will do the right thing. |
| 69 | |
| 70 | HAProxy doesn't write log files, but it relies on the standard syslog protocol |
| 71 | to send logs to a remote server (which is often located on the same system). |
| 72 | |
| 73 | HAProxy uses its internal clock to enforce timeouts, that is derived from the |
| 74 | system's time but where unexpected drift is corrected. This is done by limiting |
| 75 | the time spent waiting in poll() for an event, and measuring the time it really |
| 76 | took. In practice it never waits more than one second. This explains why, when |
| 77 | running strace over a completely idle process, periodic calls to poll() (or any |
| 78 | of its variants) surrounded by two gettimeofday() calls are noticed. They are |
| 79 | normal, completely harmless and so cheap that the load they imply is totally |
| 80 | undetectable at the system scale, so there's nothing abnormal there. Example : |
| 81 | |
| 82 | 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0 |
| 83 | 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0 |
| 84 | 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0 |
| 85 | 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0 |
| 86 | 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0 |
| 87 | 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0 |
| 88 | |
| 89 | HAProxy is a TCP proxy, not a router. It deals with established connections that |
| 90 | have been validated by the kernel, and not with packets of any form nor with |
| 91 | sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence |
| 92 | may prevent it from binding a port. It relies on the system to accept incoming |
| 93 | connections and to initiate outgoing connections. An immediate effect of this is |
| 94 | that there is no relation between packets observed on the two sides of a |
| 95 | forwarded connection, which can be of different size, numbers and even family. |
| 96 | Since a connection may only be accepted from a socket in LISTEN state, all the |
| 97 | sockets it is listening to are necessarily visible using the "netstat" utility |
| 98 | to show listening sockets. Example : |
| 99 | |
| 100 | # netstat -ltnp |
| 101 | Active Internet connections (only servers) |
| 102 | Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name |
| 103 | tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd |
| 104 | tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy |
| 105 | tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy |
| 106 | |
| 107 | |
| 108 | 3. Starting HAProxy |
| 109 | ------------------- |
| 110 | |
| 111 | HAProxy is started by invoking the "haproxy" program with a number of arguments |
| 112 | passed on the command line. The actual syntax is : |
| 113 | |
| 114 | $ haproxy [<options>]* |
| 115 | |
| 116 | where [<options>]* is any number of options. An option always starts with '-' |
| 117 | followed by one of more letters, and possibly followed by one or multiple extra |
| 118 | arguments. Without any option, HAProxy displays the help page with a reminder |
| 119 | about supported options. Available options may vary slightly based on the |
| 120 | operating system. A fair number of these options overlap with an equivalent one |
| 121 | if the "global" section. In this case, the command line always has precedence |
| 122 | over the configuration file, so that the command line can be used to quickly |
| 123 | enforce some settings without touching the configuration files. The current |
| 124 | list of options is : |
| 125 | |
| 126 | -- <cfgfile>* : all the arguments following "--" are paths to configuration |
Maxime de Roucy | 379d9c7 | 2016-05-13 23:52:56 +0200 | [diff] [blame] | 127 | file/directory to be loaded and processed in the declaration order. It is |
| 128 | mostly useful when relying on the shell to load many files that are |
| 129 | numerically ordered. See also "-f". The difference between "--" and "-f" is |
| 130 | that one "-f" must be placed before each file name, while a single "--" is |
| 131 | needed before all file names. Both options can be used together, the |
| 132 | command line ordering still applies. When more than one file is specified, |
| 133 | each file must start on a section boundary, so the first keyword of each |
| 134 | file must be one of "global", "defaults", "peers", "listen", "frontend", |
| 135 | "backend", and so on. A file cannot contain just a server list for example. |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 136 | |
Maxime de Roucy | 379d9c7 | 2016-05-13 23:52:56 +0200 | [diff] [blame] | 137 | -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be |
| 138 | loaded. If <cfgdir> is a directory, all the files (and only files) it |
Maxime de Roucy | e384139 | 2016-05-18 23:13:38 +0200 | [diff] [blame^] | 139 | containes are added in lexical order (using LC_COLLATE=C) to the list of |
Maxime de Roucy | 379d9c7 | 2016-05-13 23:52:56 +0200 | [diff] [blame] | 140 | configuration files to be loaded ; only files with ".cfg" extension are |
| 141 | added, only non hidden files (not prefixed with ".") are added. |
| 142 | Configuration files are loaded and processed in their declaration order. |
| 143 | This option may be specified multiple times to load multiple files. See |
| 144 | also "--". The difference between "--" and "-f" is that one "-f" must be |
| 145 | placed before each file name, while a single "--" is needed before all file |
| 146 | names. Both options can be used together, the command line ordering still |
| 147 | applies. When more than one file is specified, each file must start on a |
| 148 | section boundary, so the first keyword of each file must be one of |
| 149 | "global", "defaults", "peers", "listen", "frontend", "backend", and so on. |
| 150 | A file cannot contain just a server list for example. |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 151 | |
| 152 | -C <dir> : changes to directory <dir> before loading configuration |
| 153 | files. This is useful when using relative paths. Warning when using |
| 154 | wildcards after "--" which are in fact replaced by the shell before |
| 155 | starting haproxy. |
| 156 | |
| 157 | -D : start as a daemon. The process detaches from the current terminal after |
| 158 | forking, and errors are not reported anymore in the terminal. It is |
| 159 | equivalent to the "daemon" keyword in the "global" section of the |
| 160 | configuration. It is recommended to always force it in any init script so |
| 161 | that a faulty configuration doesn't prevent the system from booting. |
| 162 | |
| 163 | -Ds : work in systemd mode. Only used by the systemd wrapper. |
| 164 | |
| 165 | -L <name> : change the local peer name to <name>, which defaults to the local |
| 166 | hostname. This is used only with peers replication. |
| 167 | |
| 168 | -N <limit> : sets the default per-proxy maxconn to <limit> instead of the |
| 169 | builtin default value (usually 2000). Only useful for debugging. |
| 170 | |
| 171 | -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or |
| 172 | "quiet". |
| 173 | |
| 174 | -c : only performs a check of the configuration files and exits before trying |
| 175 | to bind. The exit status is zero if everything is OK, or non-zero if an |
| 176 | error is encountered. |
| 177 | |
| 178 | -d : enable debug mode. This disables daemon mode, forces the process to stay |
| 179 | in foreground and to show incoming and outgoing events. It is equivalent to |
| 180 | the "global" section's "debug" keyword. It must never be used in an init |
| 181 | script. |
| 182 | |
| 183 | -dG : disable use of getaddrinfo() to resolve host names into addresses. It |
| 184 | can be used when suspecting that getaddrinfo() doesn't work as expected. |
| 185 | This option was made available because many bogus implementations of |
| 186 | getaddrinfo() exist on various systems and cause anomalies that are |
| 187 | difficult to troubleshoot. |
| 188 | |
| 189 | -dM[<byte>] : forces memory poisonning, which means that each and every |
| 190 | memory region allocated with malloc() or pool_alloc2() will be filled with |
| 191 | <byte> before being passed to the caller. When <byte> is not specified, it |
| 192 | defaults to 0x50 ('P'). While this slightly slows down operations, it is |
| 193 | useful to reliably trigger issues resulting from missing initializations in |
| 194 | the code that cause random crashes. Note that -dM0 has the effect of |
| 195 | turning any malloc() into a calloc(). In any case if a bug appears or |
| 196 | disappears when using this option it means there is a bug in haproxy, so |
| 197 | please report it. |
| 198 | |
| 199 | -dS : disable use of the splice() system call. It is equivalent to the |
| 200 | "global" section's "nosplice" keyword. This may be used when splice() is |
| 201 | suspected to behave improperly or to cause performance issues, or when |
| 202 | using strace to see the forwarded data (which do not appear when using |
| 203 | splice()). |
| 204 | |
| 205 | -dV : disable SSL verify on the server side. It is equivalent to having |
| 206 | "ssl-server-verify none" in the "global" section. This is useful when |
| 207 | trying to reproduce production issues out of the production |
| 208 | environment. Never use this in an init script as it degrades SSL security |
| 209 | to the servers. |
| 210 | |
| 211 | -db : disable background mode and multi-process mode. The process remains in |
| 212 | foreground. It is mainly used during development or during small tests, as |
| 213 | Ctrl-C is enough to stop the process. Never use it in an init script. |
| 214 | |
| 215 | -de : disable the use of the "epoll" poller. It is equivalent to the "global" |
| 216 | section's keyword "noepoll". It is mostly useful when suspecting a bug |
| 217 | related to this poller. On systems supporting epoll, the fallback will |
| 218 | generally be the "poll" poller. |
| 219 | |
| 220 | -dk : disable the use of the "kqueue" poller. It is equivalent to the |
| 221 | "global" section's keyword "nokqueue". It is mostly useful when suspecting |
| 222 | a bug related to this poller. On systems supporting kqueue, the fallback |
| 223 | will generally be the "poll" poller. |
| 224 | |
| 225 | -dp : disable the use of the "poll" poller. It is equivalent to the "global" |
| 226 | section's keyword "nopoll". It is mostly useful when suspecting a bug |
| 227 | related to this poller. On systems supporting poll, the fallback will |
| 228 | generally be the "select" poller, which cannot be disabled and is limited |
| 229 | to 1024 file descriptors. |
| 230 | |
Willy Tarreau | 7006045 | 2015-12-14 12:46:07 +0100 | [diff] [blame] | 231 | -m <limit> : limit the total allocatable memory to <limit> megabytes across |
| 232 | all processes. This may cause some connection refusals or some slowdowns |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 233 | depending on the amount of memory needed for normal operations. This is |
Willy Tarreau | 7006045 | 2015-12-14 12:46:07 +0100 | [diff] [blame] | 234 | mostly used to force the processes to work in a constrained resource usage |
| 235 | scenario. It is important to note that the memory is not shared between |
| 236 | processes, so in a multi-process scenario, this value is first divided by |
| 237 | global.nbproc before forking. |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 238 | |
| 239 | -n <limit> : limits the per-process connection limit to <limit>. This is |
| 240 | equivalent to the global section's keyword "maxconn". It has precedence |
| 241 | over this keyword. This may be used to quickly force lower limits to avoid |
| 242 | a service outage on systems where resource limits are too low. |
| 243 | |
| 244 | -p <file> : write all processes' pids into <file> during startup. This is |
| 245 | equivalent to the "global" section's keyword "pidfile". The file is opened |
| 246 | before entering the chroot jail, and after doing the chdir() implied by |
| 247 | "-C". Each pid appears on its own line. |
| 248 | |
| 249 | -q : set "quiet" mode. This disables some messages during the configuration |
| 250 | parsing and during startup. It can be used in combination with "-c" to |
| 251 | just check if a configuration file is valid or not. |
| 252 | |
| 253 | -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot |
| 254 | completion to ask them to finish what they are doing and to leave. <pid> |
| 255 | is a list of pids to signal (one per argument). The list ends on any |
| 256 | option starting with a "-". It is not a problem if the list of pids is |
| 257 | empty, so that it can be built on the fly based on the result of a command |
| 258 | like "pidof" or "pgrep". |
| 259 | |
| 260 | -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after |
| 261 | boot completion to terminate them immediately without finishing what they |
| 262 | were doing. <pid> is a list of pids to signal (one per argument). The list |
| 263 | is ends on any option starting with a "-". It is not a problem if the list |
| 264 | of pids is empty, so that it can be built on the fly based on the result of |
| 265 | a command like "pidof" or "pgrep". |
| 266 | |
| 267 | -v : report the version and build date. |
| 268 | |
| 269 | -vv : display the version, build options, libraries versions and usable |
| 270 | pollers. This output is systematically requested when filing a bug report. |
| 271 | |
| 272 | A safe way to start HAProxy from an init file consists in forcing the deamon |
| 273 | mode, storing existing pids to a pid file and using this pid file to notify |
| 274 | older processes to finish before leaving : |
| 275 | |
| 276 | haproxy -f /etc/haproxy.cfg \ |
| 277 | -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) |
| 278 | |
| 279 | When the configuration is split into a few specific files (eg: tcp vs http), |
| 280 | it is recommended to use the "-f" option : |
| 281 | |
| 282 | haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \ |
| 283 | -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \ |
| 284 | -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \ |
| 285 | -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) |
| 286 | |
| 287 | When an unknown number of files is expected, such as customer-specific files, |
| 288 | it is recommended to assign them a name starting with a fixed-size sequence |
| 289 | number and to use "--" to load them, possibly after loading some defaults : |
| 290 | |
| 291 | haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \ |
| 292 | -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \ |
| 293 | -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \ |
| 294 | -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \ |
| 295 | -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/* |
| 296 | |
| 297 | Sometimes a failure to start may happen for whatever reason. Then it is |
| 298 | important to verify if the version of HAProxy you are invoking is the expected |
| 299 | version and if it supports the features you are expecting (eg: SSL, PCRE, |
| 300 | compression, Lua, etc). This can be verified using "haproxy -vv". Some |
| 301 | important information such as certain build options, the target system and |
| 302 | the versions of the libraries being used are reported there. It is also what |
| 303 | you will systematically be asked for when posting a bug report : |
| 304 | |
| 305 | $ haproxy -vv |
| 306 | HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08 |
| 307 | Copyright 2000-2015 Willy Tarreau <willy@haproxy.org> |
| 308 | |
| 309 | Build options : |
| 310 | TARGET = linux2628 |
| 311 | CPU = generic |
| 312 | CC = gcc |
| 313 | CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \ |
| 314 | -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19 |
| 315 | OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 |
| 316 | |
| 317 | Default settings : |
| 318 | maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200 |
| 319 | |
| 320 | Encrypted password support via crypt(3): yes |
| 321 | Built with zlib version : 1.2.6 |
| 322 | Compression algorithms supported : identity("identity"), deflate("deflate"), \ |
| 323 | raw-deflate("deflate"), gzip("gzip") |
| 324 | Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015 |
| 325 | Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015 |
| 326 | OpenSSL library supports TLS extensions : yes |
| 327 | OpenSSL library supports SNI : yes |
| 328 | OpenSSL library supports prefer-server-ciphers : yes |
| 329 | Built with PCRE version : 8.12 2011-01-15 |
| 330 | PCRE library supports JIT : no (USE_PCRE_JIT not set) |
| 331 | Built with Lua version : Lua 5.3.1 |
| 332 | Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND |
| 333 | |
| 334 | Available polling systems : |
| 335 | epoll : pref=300, test result OK |
| 336 | poll : pref=200, test result OK |
| 337 | select : pref=150, test result OK |
| 338 | Total: 3 (3 usable), will use epoll. |
| 339 | |
| 340 | The relevant information that many non-developer users can verify here are : |
| 341 | - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit |
| 342 | ID "a088d3" which is the 4th one after after official version "1.6-dev7". |
| 343 | Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in |
| 344 | fact "1.6-dev7". This is the 7th development version of what will become |
| 345 | version 1.6 in the future. A development version not suitable for use in |
| 346 | production (unless you know exactly what you are doing). A stable version |
| 347 | will show as a 3-numbers version, such as "1.5.14-16f863", indicating the |
| 348 | 14th level of fix on top of version 1.5. This is a production-ready version. |
| 349 | |
| 350 | - the release date : 2015/10/08. It is represented in the universal |
| 351 | year/month/day format. Here this means August 8th, 2015. Given that stable |
| 352 | releases are issued every few months (1-2 months at the beginning, sometimes |
| 353 | 6 months once the product becomes very stable), if you're seeing an old date |
| 354 | here, it means you're probably affected by a number of bugs or security |
| 355 | issues that have since been fixed and that it might be worth checking on the |
| 356 | official site. |
| 357 | |
| 358 | - build options : they are relevant to people who build their packages |
| 359 | themselves, they can explain why things are not behaving as expected. For |
| 360 | example the development version above was built for Linux 2.6.28 or later, |
| 361 | targetting a generic CPU (no CPU-specific optimizations), and lacks any |
| 362 | code optimization (-O0) so it will perform poorly in terms of performance. |
| 363 | |
| 364 | - libraries versions : zlib version is reported as found in the library |
| 365 | itself. In general zlib is considered a very stable product and upgrades |
| 366 | are almost never needed. OpenSSL reports two versions, the version used at |
| 367 | build time and the one being used, as found on the system. These ones may |
| 368 | differ by the last letter but never by the numbers. The build date is also |
| 369 | reported because most OpenSSL bugs are security issues and need to be taken |
| 370 | seriously, so this library absolutely needs to be kept up to date. Seeing a |
| 371 | 4-months old version here is highly suspicious and indeed an update was |
| 372 | missed. PCRE provides very fast regular expressions and is highly |
| 373 | recommended. Certain of its extensions such as JIT are not present in all |
| 374 | versions and still young so some people prefer not to build with them, |
| 375 | which is why the biuld status is reported as well. Regarding the Lua |
| 376 | scripting language, HAProxy expects version 5.3 which is very young since |
| 377 | it was released a little time before HAProxy 1.6. It is important to check |
| 378 | on the Lua web site if some fixes are proposed for this branch. |
| 379 | |
| 380 | - Available polling systems will affect the process's scalability when |
| 381 | dealing with more than about one thousand of concurrent connections. These |
| 382 | ones are only available when the correct system was indicated in the TARGET |
| 383 | variable during the build. The "epoll" mechanism is highly recommended on |
| 384 | Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them |
| 385 | will result in poll() or even select() being used, causing a high CPU usage |
| 386 | when dealing with a lot of connections. |
| 387 | |
| 388 | |
| 389 | 4. Stopping and restarting HAProxy |
| 390 | ---------------------------------- |
| 391 | |
| 392 | HAProxy supports a graceful and a hard stop. The hard stop is simple, when the |
| 393 | SIGTERM signal is sent to the haproxy process, it immediately quits and all |
| 394 | established connections are closed. The graceful stop is triggered when the |
| 395 | SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding |
| 396 | from listening ports, but continue to process existing connections until they |
| 397 | close. Once the last connection is closed, the process leaves. |
| 398 | |
| 399 | The hard stop method is used for the "stop" or "restart" actions of the service |
| 400 | management script. The graceful stop is used for the "reload" action which |
| 401 | tries to seamlessly reload a new configuration in a new process. |
| 402 | |
| 403 | Both of these signals may be sent by the new haproxy process itself during a |
| 404 | reload or restart, so that they are sent at the latest possible moment and only |
| 405 | if absolutely required. This is what is performed by the "-st" (hard) and "-sf" |
| 406 | (graceful) options respectively. |
| 407 | |
| 408 | To understand better how these signals are used, it is important to understand |
| 409 | the whole restart mechanism. |
| 410 | |
| 411 | First, an existing haproxy process is running. The administrator uses a system |
| 412 | specific command such as "/etc/init.d/haproxy reload" to indicate he wants to |
| 413 | take the new configuration file into effect. What happens then is the following. |
| 414 | First, the service script (/etc/init.d/haproxy or equivalent) will verify that |
| 415 | the configuration file parses correctly using "haproxy -c". After that it will |
| 416 | try to start haproxy with this configuration file, using "-st" or "-sf". |
| 417 | |
| 418 | Then HAProxy tries to bind to all listening ports. If some fatal errors happen |
| 419 | (eg: address not present on the system, permission denied), the process quits |
| 420 | with an error. If a socket binding fails because a port is already in use, then |
| 421 | the process will first send a SIGTTOU signal to all the pids specified in the |
| 422 | "-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs |
| 423 | all existing haproxy processes to temporarily stop listening to their ports so |
| 424 | that the new process can try to bind again. During this time, the old process |
| 425 | continues to process existing connections. If the binding still fails (because |
| 426 | for example a port is shared with another daemon), then the new process sends a |
| 427 | SIGTTIN signal to the old processes to instruct them to resume operations just |
| 428 | as if nothing happened. The old processes will then restart listening to the |
| 429 | ports and continue to accept connections. Not that this mechanism is system |
| 430 | dependant and some operating systems may not support it in multi-process mode. |
| 431 | |
| 432 | If the new process manages to bind correctly to all ports, then it sends either |
| 433 | the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case |
| 434 | of "-sf") to all processes to notify them that it is now in charge of operations |
| 435 | and that the old processes will have to leave, either immediately or once they |
| 436 | have finished their job. |
| 437 | |
| 438 | It is important to note that during this timeframe, there are two small windows |
| 439 | of a few milliseconds each where it is possible that a few connection failures |
| 440 | will be noticed during high loads. Typically observed failure rates are around |
| 441 | 1 failure during a reload operation every 10000 new connections per second, |
| 442 | which means that a heavily loaded site running at 30000 new connections per |
| 443 | second may see about 3 failed connection upon every reload. The two situations |
| 444 | where this happens are : |
| 445 | |
| 446 | - if the new process fails to bind due to the presence of the old process, |
| 447 | it will first have to go through the SIGTTOU+SIGTTIN sequence, which |
| 448 | typically lasts about one millisecond for a few tens of frontends, and |
| 449 | during which some ports will not be bound to the old process and not yet |
| 450 | bound to the new one. HAProxy works around this on systems that support the |
| 451 | SO_REUSEPORT socket options, as it allows the new process to bind without |
| 452 | first asking the old one to unbind. Most BSD systems have been supporting |
| 453 | this almost forever. Linux has been supporting this in version 2.0 and |
| 454 | dropped it around 2.2, but some patches were floating around by then. It |
| 455 | was reintroduced in kernel 3.9, so if you are observing a connection |
| 456 | failure rate above the one mentionned above, please ensure that your kernel |
| 457 | is 3.9 or newer, or that relevant patches were backported to your kernel |
| 458 | (less likely). |
| 459 | |
| 460 | - when the old processes close the listening ports, the kernel may not always |
| 461 | redistribute any pending connection that was remaining in the socket's |
| 462 | backlog. Under high loads, a SYN packet may happen just before the socket |
| 463 | is closed, and will lead to an RST packet being sent to the client. In some |
| 464 | critical environments where even one drop is not acceptable, these ones are |
| 465 | sometimes dealt with using firewall rules to block SYN packets during the |
| 466 | reload, forcing the client to retransmit. This is totally system-dependent, |
| 467 | as some systems might be able to visit other listening queues and avoid |
| 468 | this RST. A second case concerns the ACK from the client on a local socket |
| 469 | that was in SYN_RECV state just before the close. This ACK will lead to an |
| 470 | RST packet while the haproxy process is still not aware of it. This one is |
| 471 | harder to get rid of, though the firewall filtering rules mentionned above |
| 472 | will work well if applied one second or so before restarting the process. |
| 473 | |
| 474 | For the vast majority of users, such drops will never ever happen since they |
| 475 | don't have enough load to trigger the race conditions. And for most high traffic |
| 476 | users, the failure rate is still fairly within the noise margin provided that at |
| 477 | least SO_REUSEPORT is properly supported on their systems. |
| 478 | |
| 479 | |
| 480 | 5. File-descriptor limitations |
| 481 | ------------------------------ |
| 482 | |
| 483 | In order to ensure that all incoming connections will successfully be served, |
| 484 | HAProxy computes at load time the total number of file descriptors that will be |
| 485 | needed during the process's life. A regular Unix process is generally granted |
| 486 | 1024 file descriptors by default, and a privileged process can raise this limit |
| 487 | itself. This is one reason for starting HAProxy as root and letting it adjust |
| 488 | the limit. The default limit of 1024 file descriptors roughly allow about 500 |
| 489 | concurrent connections to be processed. The computation is based on the global |
| 490 | maxconn parameter which limits the total number of connections per process, the |
| 491 | number of listeners, the number of servers which have a health check enabled, |
| 492 | the agent checks, the peers, the loggers and possibly a few other technical |
| 493 | requirements. A simple rough estimate of this number consists in simply |
| 494 | doubling the maxconn value and adding a few tens to get the approximate number |
| 495 | of file descriptors needed. |
| 496 | |
| 497 | Originally HAProxy did not know how to compute this value, and it was necessary |
| 498 | to pass the value using the "ulimit-n" setting in the global section. This |
| 499 | explains why even today a lot of configurations are seen with this setting |
| 500 | present. Unfortunately it was often miscalculated resulting in connection |
| 501 | failures when approaching maxconn instead of throttling incoming connection |
| 502 | while waiting for the needed resources. For this reason it is important to |
| 503 | remove any vestigal "ulimit-n" setting that can remain from very old versions. |
| 504 | |
| 505 | Raising the number of file descriptors to accept even moderate loads is |
| 506 | mandatory but comes with some OS-specific adjustments. First, the select() |
| 507 | polling system is limited to 1024 file descriptors. In fact on Linux it used |
| 508 | to be capable of handling more but since certain OS ship with excessively |
| 509 | restrictive SELinux policies forbidding the use of select() with more than |
| 510 | 1024 file descriptors, HAProxy now refuses to start in this case in order to |
| 511 | avoid any issue at run time. On all supported operating systems, poll() is |
| 512 | available and will not suffer from this limitation. It is automatically picked |
| 513 | so there is nothing ot do to get a working configuration. But poll's becomes |
| 514 | very slow when the number of file descriptors increases. While HAProxy does its |
| 515 | best to limit this performance impact (eg: via the use of the internal file |
| 516 | descriptor cache and batched processing), a good rule of thumb is that using |
| 517 | poll() with more than a thousand concurrent connections will use a lot of CPU. |
| 518 | |
| 519 | For Linux systems base on kernels 2.6 and above, the epoll() system call will |
| 520 | be used. It's a much more scalable mechanism relying on callbacks in the kernel |
| 521 | that guarantee a constant wake up time regardless of the number of registered |
| 522 | monitored file descriptors. It is automatically used where detected, provided |
| 523 | that HAProxy had been built for one of the Linux flavors. Its presence and |
| 524 | support can be verified using "haproxy -vv". |
| 525 | |
| 526 | For BSD systems which support it, kqueue() is available as an alternative. It |
| 527 | is much faster than poll() and even slightly faster than epoll() thanks to its |
| 528 | batched handling of changes. At least FreeBSD and OpenBSD support it. Just like |
| 529 | with Linux's epoll(), its support and availability are reported in the output |
| 530 | of "haproxy -vv". |
| 531 | |
| 532 | Having a good poller is one thing, but it is mandatory that the process can |
| 533 | reach the limits. When HAProxy starts, it immediately sets the new process's |
| 534 | file descriptor limits and verifies if it succeeds. In case of failure, it |
| 535 | reports it before forking so that the administrator can see the problem. As |
| 536 | long as the process is started by as root, there should be no reason for this |
| 537 | setting to fail. However, it can fail if the process is started by an |
| 538 | unprivileged user. If there is a compelling reason for *not* starting haproxy |
| 539 | as root (eg: started by end users, or by a per-application account), then the |
| 540 | file descriptor limit can be raised by the system administrator for this |
| 541 | specific user. The effectiveness of the setting can be verified by issuing |
| 542 | "ulimit -n" from the user's command line. It should reflect the new limit. |
| 543 | |
| 544 | Warning: when an unprivileged user's limits are changed in this user's account, |
| 545 | it is fairly common that these values are only considered when the user logs in |
| 546 | and not at all in some scripts run at system boot time nor in crontabs. This is |
| 547 | totally dependent on the operating system, keep in mind to check "ulimit -n" |
| 548 | before starting haproxy when running this way. The general advice is never to |
| 549 | start haproxy as an unprivileged user for production purposes. Another good |
| 550 | reason is that it prevents haproxy from enabling some security protections. |
| 551 | |
| 552 | Once it is certain that the system will allow the haproxy process to use the |
| 553 | requested number of file descriptors, two new system-specific limits may be |
| 554 | encountered. The first one is the system-wide file descriptor limit, which is |
| 555 | the total number of file descriptors opened on the system, covering all |
| 556 | processes. When this limit is reached, accept() or socket() will typically |
| 557 | return ENFILE. The second one is the per-process hard limit on the number of |
| 558 | file descriptors, it prevents setrlimit() from being set higher. Both are very |
| 559 | dependent on the operating system. On Linux, the system limit is set at boot |
| 560 | based on the amount of memory. It can be changed with the "fs.file-max" sysctl. |
| 561 | And the per-process hard limit is set to 1048576 by default, but it can be |
| 562 | changed using the "fs.nr_open" sysctl. |
| 563 | |
| 564 | File descriptor limitations may be observed on a running process when they are |
| 565 | set too low. The strace utility will report that accept() and socket() return |
| 566 | "-1 EMFILE" when the process's limits have been reached. In this case, simply |
| 567 | raising the "ulimit-n" value (or removing it) will solve the problem. If these |
| 568 | system calls return "-1 ENFILE" then it means that the kernel's limits have |
| 569 | been reached and that something must be done on a system-wide parameter. These |
| 570 | trouble must absolutely be addressed, as they result in high CPU usage (when |
| 571 | accept() fails) and failed connections that are generally visible to the user. |
| 572 | One solution also consists in lowering the global maxconn value to enforce |
| 573 | serialization, and possibly to disable HTTP keep-alive to force connections |
| 574 | to be released and reused faster. |
| 575 | |
| 576 | |
| 577 | 6. Memory management |
| 578 | -------------------- |
| 579 | |
| 580 | HAProxy uses a simple and fast pool-based memory management. Since it relies on |
| 581 | a small number of different object types, it's much more efficient to pick new |
| 582 | objects from a pool which already contains objects of the appropriate size than |
| 583 | to call malloc() for each different size. The pools are organized as a stack or |
| 584 | LIFO, so that newly allocated objects are taken from recently released objects |
| 585 | still hot in the CPU caches. Pools of similar sizes are merged together, in |
| 586 | order to limit memory fragmentation. |
| 587 | |
| 588 | By default, since the focus is set on performance, each released object is put |
| 589 | back into the pool it came from, and allocated objects are never freed since |
| 590 | they are expected to be reused very soon. |
| 591 | |
| 592 | On the CLI, it is possible to check how memory is being used in pools thanks to |
| 593 | the "show pools" command : |
| 594 | |
| 595 | > show pools |
| 596 | Dumping pools usage. Use SIGQUIT to flush them. |
| 597 | - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED] |
| 598 | - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] |
| 599 | - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED] |
| 600 | - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED] |
| 601 | - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED] |
| 602 | - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] |
| 603 | - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED] |
| 604 | - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] |
| 605 | - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED] |
| 606 | - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED] |
| 607 | - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED] |
| 608 | Total: 11 pools, 26608 bytes allocated, 18544 used. |
| 609 | |
| 610 | The pool name is only indicative, it's the name of the first object type using |
| 611 | this pool. The size in parenthesis is the object size for objects in this pool. |
| 612 | Object sizes are always rounded up to the closest multiple of 16 bytes. The |
| 613 | number of objects currently allocated and the equivalent number of bytes is |
| 614 | reported so that it is easy to know which pool is responsible for the highest |
| 615 | memory usage. The number of objects currently in use is reported as well in the |
| 616 | "used" field. The difference between "allocated" and "used" corresponds to the |
| 617 | objects that have been freed and are available for immediate use. |
| 618 | |
| 619 | It is possible to limit the amount of memory allocated per process using the |
| 620 | "-m" command line option, followed by a number of megabytes. It covers all of |
| 621 | the process's addressable space, so that includes memory used by some libraries |
| 622 | as well as the stack, but it is a reliable limit when building a resource |
| 623 | constrained system. It works the same way as "ulimit -v" on systems which have |
| 624 | it, or "ulimit -d" for the other ones. |
| 625 | |
| 626 | If a memory allocation fails due to the memory limit being reached or because |
| 627 | the system doesn't have any enough memory, then haproxy will first start to |
| 628 | free all available objects from all pools before attempting to allocate memory |
| 629 | again. This mechanism of releasing unused memory can be triggered by sending |
| 630 | the signal SIGQUIT to the haproxy process. When doing so, the pools state prior |
| 631 | to the flush will also be reported to stderr when the process runs in |
| 632 | foreground. |
| 633 | |
| 634 | During a reload operation, the process switched to the graceful stop state also |
| 635 | automatically performs some flushes after releasing any connection so that all |
| 636 | possible memory is released to save it for the new process. |
| 637 | |
| 638 | |
| 639 | 7. CPU usage |
| 640 | ------------ |
| 641 | |
| 642 | HAProxy normally spends most of its time in the system and a smaller part in |
| 643 | userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end |
| 644 | connection setups and closes per second at 100% CPU on a single core. When one |
| 645 | core is saturated, typical figures are : |
| 646 | - 95% system, 5% user for long TCP connections or large HTTP objects |
| 647 | - 85% system and 15% user for short TCP connections or small HTTP objects in |
| 648 | close mode |
| 649 | - 70% system and 30% user for small HTTP objects in keep-alive mode |
| 650 | |
| 651 | The amount of rules processing and regular expressions will increase the user |
| 652 | land part. The presence of firewall rules, connection tracking, complex routing |
| 653 | tables in the system will instead increase the system part. |
| 654 | |
| 655 | On most systems, the CPU time observed during network transfers can be cut in 4 |
| 656 | parts : |
| 657 | - the interrupt part, which concerns all the processing performed upon I/O |
| 658 | receipt, before the target process is even known. Typically Rx packets are |
| 659 | accounted for in interrupt. On some systems such as Linux where interrupt |
| 660 | processing may be deferred to a dedicated thread, it can appear as softirq, |
| 661 | and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of |
| 662 | this load is generally defined by the hardware settings, though in the case |
| 663 | of softirq it is often possible to remap the processing to another CPU. |
| 664 | This interrupt part will often be perceived as parasitic since it's not |
| 665 | associated with any process, but it actually is some processing being done |
| 666 | to prepare the work for the process. |
| 667 | |
| 668 | - the system part, which concerns all the processing done using kernel code |
| 669 | called from userland. System calls are accounted as system for example. All |
| 670 | synchronously delivered Tx packets will be accounted for as system time. If |
| 671 | some packets have to be deferred due to queues filling up, they may then be |
| 672 | processed in interrupt context later (eg: upon receipt of an ACK opening a |
| 673 | TCP window). |
| 674 | |
| 675 | - the user part, which exclusively runs application code in userland. HAProxy |
| 676 | runs exclusively in this part, though it makes heavy use of system calls. |
| 677 | Rules processing, regular expressions, compression, encryption all add to |
| 678 | the user portion of CPU consumption. |
| 679 | |
| 680 | - the idle part, which is what the CPU does when there is nothing to do. For |
| 681 | example HAProxy waits for an incoming connection, or waits for some data to |
| 682 | leave, meaning the system is waiting for an ACK from the client to push |
| 683 | these data. |
| 684 | |
| 685 | In practice regarding HAProxy's activity, it is in general reasonably accurate |
| 686 | (but totally inexact) to consider that interrupt/softirq are caused by Rx |
| 687 | processing in kernel drivers, that user-land is caused by layer 7 processing |
| 688 | in HAProxy, and that system time is caused by network processing on the Tx |
| 689 | path. |
| 690 | |
| 691 | Since HAProxy runs around an event loop, it waits for new events using poll() |
| 692 | (or any alternative) and processes all these events as fast as possible before |
| 693 | going back to poll() waiting for new events. It measures the time spent waiting |
| 694 | in poll() compared to the time spent doing processing events. The ratio of |
| 695 | polling time vs total time is called the "idle" time, it's the amount of time |
| 696 | spent waiting for something to happen. This ratio is reported in the stats page |
| 697 | on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means |
| 698 | the load is extremely low. When it's close to 0%, it means that there is |
| 699 | constantly some activity. While it cannot be very accurate on an overloaded |
| 700 | system due to other processes possibly preempting the CPU from the haproxy |
| 701 | process, it still provides a good estimate about how HAProxy considers it is |
| 702 | working : if the load is low and the idle ratio is low as well, it may indicate |
| 703 | that HAProxy has a lot of work to do, possibly due to very expensive rules that |
| 704 | have to be processed. Conversely, if HAProxy indicates the idle is close to |
| 705 | 100% while things are slow, it means that it cannot do anything to speed things |
| 706 | up because it is already waiting for incoming data to process. In the example |
| 707 | below, haproxy is completely idle : |
| 708 | |
| 709 | $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle |
| 710 | Idle_pct: 100 |
| 711 | |
| 712 | When the idle ratio starts to become very low, it is important to tune the |
| 713 | system and place processes and interrupts correctly to save the most possible |
| 714 | CPU resources for all tasks. If a firewall is present, it may be worth trying |
| 715 | to disable it or to tune it to ensure it is not responsible for a large part |
| 716 | of the performance limitation. It's worth noting that unloading a stateful |
| 717 | firewall generally reduces both the amount of interrupt/softirq and of system |
| 718 | usage since such firewalls act both on the Rx and the Tx paths. On Linux, |
| 719 | unloading the nf_conntrack and ip_conntrack modules will show whether there is |
| 720 | anything to gain. If so, then the module runs with default settings and you'll |
| 721 | have to figure how to tune it for better performance. In general this consists |
| 722 | in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will |
| 723 | disable the "pf" firewall and its stateful engine at the same time. |
| 724 | |
| 725 | If it is observed that a lot of time is spent in interrupt/softirq, it is |
| 726 | important to ensure that they don't run on the same CPU. Most systems tend to |
| 727 | pin the tasks on the CPU where they receive the network traffic because for |
| 728 | certain workloads it improves things. But with heavily network-bound workloads |
| 729 | it is the opposite as the haproxy process will have to fight against its kernel |
| 730 | counterpart. Pinning haproxy to one CPU core and the interrupts to another one, |
| 731 | all sharing the same L3 cache tends to sensibly increase network performance |
| 732 | because in practice the amount of work for haproxy and the network stack are |
| 733 | quite close, so they can almost fill an entire CPU each. On Linux this is done |
| 734 | using taskset (for haproxy) or using cpu-map (from the haproxy config), and the |
| 735 | interrupts are assigned under /proc/irq. Many network interfaces support |
| 736 | multiple queues and multiple interrupts. In general it helps to spread them |
| 737 | across a small number of CPU cores provided they all share the same L3 cache. |
| 738 | Please always stop irq_balance which always does the worst possible thing on |
| 739 | such workloads. |
| 740 | |
| 741 | For CPU-bound workloads consisting in a lot of SSL traffic or a lot of |
| 742 | compression, it may be worth using multiple processes dedicated to certain |
| 743 | tasks, though there is no universal rule here and experimentation will have to |
| 744 | be performed. |
| 745 | |
| 746 | In order to increase the CPU capacity, it is possible to make HAProxy run as |
| 747 | several processes, using the "nbproc" directive in the global section. There |
| 748 | are some limitations though : |
| 749 | - health checks are run per process, so the target servers will get as many |
| 750 | checks as there are running processes ; |
| 751 | - maxconn values and queues are per-process so the correct value must be set |
| 752 | to avoid overloading the servers ; |
| 753 | - outgoing connections should avoid using port ranges to avoid conflicts |
| 754 | - stick-tables are per process and are not shared between processes ; |
| 755 | - each peers section may only run on a single process at a time ; |
| 756 | - the CLI operations will only act on a single process at a time. |
| 757 | |
| 758 | With this in mind, it appears that the easiest setup often consists in having |
| 759 | one first layer running on multiple processes and in charge for the heavy |
| 760 | processing, passing the traffic to a second layer running in a single process. |
| 761 | This mechanism is suited to SSL and compression which are the two CPU-heavy |
| 762 | features. Instances can easily be chained over UNIX sockets (which are cheaper |
fengpeiyuan | cc123c6 | 2016-01-15 16:40:53 +0800 | [diff] [blame] | 763 | than TCP sockets and which do not waste ports), and the proxy protocol which is |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 764 | useful to pass client information to the next stage. When doing so, it is |
| 765 | generally a good idea to bind all the single-process tasks to process number 1 |
| 766 | and extra tasks to next processes, as this will make it easier to generate |
| 767 | similar configurations for different machines. |
| 768 | |
| 769 | On Linux versions 3.9 and above, running HAProxy in multi-process mode is much |
| 770 | more efficient when each process uses a distinct listening socket on the same |
| 771 | IP:port ; this will make the kernel evenly distribute the load across all |
| 772 | processes instead of waking them all up. Please check the "process" option of |
| 773 | the "bind" keyword lines in the configuration manual for more information. |
| 774 | |
| 775 | |
| 776 | 8. Logging |
| 777 | ---------- |
| 778 | |
| 779 | For logging, HAProxy always relies on a syslog server since it does not perform |
| 780 | any file-system access. The standard way of using it is to send logs over UDP |
| 781 | to the log server (by default on port 514). Very commonly this is configured to |
| 782 | 127.0.0.1 where the local syslog daemon is running, but it's also used over the |
| 783 | network to log to a central server. The central server provides additional |
| 784 | benefits especially in active-active scenarios where it is desirable to keep |
| 785 | the logs merged in arrival order. HAProxy may also make use of a UNIX socket to |
| 786 | send its logs to the local syslog daemon, but it is not recommended at all, |
| 787 | because if the syslog server is restarted while haproxy runs, the socket will |
| 788 | be replaced and new logs will be lost. Since HAProxy will be isolated inside a |
| 789 | chroot jail, it will not have the ability to reconnect to the new socket. It |
| 790 | has also been observed in field that the log buffers in use on UNIX sockets are |
| 791 | very small and lead to lost messages even at very light loads. But this can be |
| 792 | fine for testing however. |
| 793 | |
| 794 | It is recommended to add the following directive to the "global" section to |
| 795 | make HAProxy log to the local daemon using facility "local0" : |
| 796 | |
| 797 | log 127.0.0.1:514 local0 |
| 798 | |
| 799 | and then to add the following one to each "defaults" section or to each frontend |
| 800 | and backend section : |
| 801 | |
| 802 | log global |
| 803 | |
| 804 | This way, all logs will be centralized through the global definition of where |
| 805 | the log server is. |
| 806 | |
| 807 | Some syslog daemons do not listen to UDP traffic by default, so depending on |
| 808 | the daemon being used, the syntax to enable this will vary : |
| 809 | |
| 810 | - on sysklogd, you need to pass argument "-r" on the daemon's command line |
| 811 | so that it listens to a UDP socket for "remote" logs ; note that there is |
| 812 | no way to limit it to address 127.0.0.1 so it will also receive logs from |
| 813 | remote systems ; |
| 814 | |
| 815 | - on rsyslogd, the following lines must be added to the configuration file : |
| 816 | |
| 817 | $ModLoad imudp |
| 818 | $UDPServerAddress * |
| 819 | $UDPServerRun 514 |
| 820 | |
| 821 | - on syslog-ng, a new source can be created the following way, it then needs |
| 822 | to be added as a valid source in one of the "log" directives : |
| 823 | |
| 824 | source s_udp { |
| 825 | udp(ip(127.0.0.1) port(514)); |
| 826 | }; |
| 827 | |
| 828 | Please consult your syslog daemon's manual for more information. If no logs are |
| 829 | seen in the system's log files, please consider the following tests : |
| 830 | |
| 831 | - restart haproxy. Each frontend and backend logs one line indicating it's |
| 832 | starting. If these logs are received, it means logs are working. |
| 833 | |
| 834 | - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some |
| 835 | activity that you expect to be logged. You should see the log messages |
| 836 | being sent using sendmsg() there. If they don't appear, restart using |
| 837 | strace on top of haproxy. If you still see no logs, it definitely means |
| 838 | that something is wrong in your configuration. |
| 839 | |
| 840 | - run tcpdump to watch for port 514, for example on the loopback interface if |
| 841 | the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the |
| 842 | packets are seen there, it's the proof they're sent then the syslogd daemon |
| 843 | needs to be troubleshooted. |
| 844 | |
| 845 | While traffic logs are sent from the frontends (where the incoming connections |
| 846 | are accepted), backends also need to be able to send logs in order to report a |
| 847 | server state change consecutive to a health check. Please consult HAProxy's |
| 848 | configuration manual for more information regarding all possible log settings. |
| 849 | |
| 850 | It is convenient to chose a facility that is not used by other deamons. HAProxy |
| 851 | examples often suggest "local0" for traffic logs and "local1" for admin logs |
| 852 | because they're never seen in field. A single facility would be enough as well. |
| 853 | Having separate logs is convenient for log analysis, but it's also important to |
| 854 | remember that logs may sometimes convey confidential information, and as such |
| 855 | they must not be mixed with other logs that may accidently be handed out to |
| 856 | unauthorized people. |
| 857 | |
| 858 | For in-field troubleshooting without impacting the server's capacity too much, |
| 859 | it is recommended to make use of the "halog" utility provided with HAProxy. |
| 860 | This is sort of a grep-like utility designed to process HAProxy log files at |
| 861 | a very fast data rate. Typical figures range between 1 and 2 GB of logs per |
| 862 | second. It is capable of extracting only certain logs (eg: search for some |
| 863 | classes of HTTP status codes, connection termination status, search by response |
| 864 | time ranges, look for errors only), count lines, limit the output to a number |
| 865 | of lines, and perform some more advanced statistics such as sorting servers |
| 866 | by response time or error counts, sorting URLs by time or count, sorting client |
| 867 | addresses by access count, and so on. It is pretty convenient to quickly spot |
| 868 | anomalies such as a bot looping on the site, and block them. |
| 869 | |
| 870 | |
| 871 | 9. Statistics and monitoring |
| 872 | ---------------------------- |
| 873 | |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 874 | It is possible to query HAProxy about its status. The most commonly used |
| 875 | mechanism is the HTTP statistics page. This page also exposes an alternative |
| 876 | CSV output format for monitoring tools. The same format is provided on the |
| 877 | Unix socket. |
| 878 | |
| 879 | |
| 880 | 9.1. CSV format |
| 881 | --------------- |
| 882 | |
| 883 | The statistics may be consulted either from the unix socket or from the HTTP |
| 884 | page. Both means provide a CSV format whose fields follow. The first line |
| 885 | begins with a sharp ('#') and has one word per comma-delimited field which |
| 886 | represents the title of the column. All other lines starting at the second one |
| 887 | use a classical CSV format using a comma as the delimiter, and the double quote |
| 888 | ('"') as an optional text delimiter, but only if the enclosed text is ambiguous |
| 889 | (if it contains a quote or a comma). The double-quote character ('"') in the |
| 890 | text is doubled ('""'), which is the format that most tools recognize. Please |
| 891 | do not insert any column before these ones in order not to break tools which |
| 892 | use hard-coded column positions. |
| 893 | |
| 894 | In brackets after each field name are the types which may have a value for |
| 895 | that field. The types are L (Listeners), F (Frontends), B (Backends), and |
| 896 | S (Servers). |
| 897 | |
| 898 | 0. pxname [LFBS]: proxy name |
| 899 | 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend, |
| 900 | any name for server/listener) |
| 901 | 2. qcur [..BS]: current queued requests. For the backend this reports the |
| 902 | number queued without a server assigned. |
| 903 | 3. qmax [..BS]: max value of qcur |
| 904 | 4. scur [LFBS]: current sessions |
| 905 | 5. smax [LFBS]: max sessions |
| 906 | 6. slim [LFBS]: configured session limit |
Willy Tarreau | c73810f | 2016-01-11 13:52:04 +0100 | [diff] [blame] | 907 | 7. stot [LFBS]: cumulative number of sessions |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 908 | 8. bin [LFBS]: bytes in |
| 909 | 9. bout [LFBS]: bytes out |
| 910 | 10. dreq [LFB.]: requests denied because of security concerns. |
| 911 | - For tcp this is because of a matched tcp-request content rule. |
| 912 | - For http this is because of a matched http-request or tarpit rule. |
| 913 | 11. dresp [LFBS]: responses denied because of security concerns. |
| 914 | - For http this is because of a matched http-request rule, or |
| 915 | "option checkcache". |
| 916 | 12. ereq [LF..]: request errors. Some of the possible causes are: |
| 917 | - early termination from the client, before the request has been sent. |
| 918 | - read error from the client |
| 919 | - client timeout |
| 920 | - client closed connection |
| 921 | - various bad requests from the client. |
| 922 | - request was tarpitted. |
| 923 | 13. econ [..BS]: number of requests that encountered an error trying to |
| 924 | connect to a backend server. The backend stat is the sum of the stat |
| 925 | for all servers of that backend, plus any connection errors not |
| 926 | associated with a particular server (such as the backend having no |
| 927 | active servers). |
| 928 | 14. eresp [..BS]: response errors. srv_abrt will be counted here also. |
| 929 | Some other errors are: |
| 930 | - write error on the client socket (won't be counted for the server stat) |
| 931 | - failure applying filters to the response. |
| 932 | 15. wretr [..BS]: number of times a connection to a server was retried. |
| 933 | 16. wredis [..BS]: number of times a request was redispatched to another |
| 934 | server. The server value counts the number of times that server was |
| 935 | switched away from. |
| 936 | 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)...) |
| 937 | 18. weight [..BS]: total weight (backend), server weight (server) |
| 938 | 19. act [..BS]: number of active servers (backend), server is active (server) |
| 939 | 20. bck [..BS]: number of backup servers (backend), server is backup (server) |
| 940 | 21. chkfail [...S]: number of failed checks. (Only counts checks failed when |
| 941 | the server is up.) |
| 942 | 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts |
| 943 | transitions to the whole backend being down, rather than the sum of the |
| 944 | counters for each server. |
| 945 | 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition |
| 946 | 24. downtime [..BS]: total downtime (in seconds). The value for the backend |
| 947 | is the downtime for the whole backend, not the sum of the server downtime. |
| 948 | 25. qlimit [...S]: configured maxqueue for the server, or nothing in the |
| 949 | value is 0 (default, meaning no limit) |
| 950 | 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...) |
| 951 | 27. iid [LFBS]: unique proxy id |
| 952 | 28. sid [L..S]: server id (unique inside a proxy) |
| 953 | 29. throttle [...S]: current throttle percentage for the server, when |
| 954 | slowstart is active, or no value if not in slowstart. |
| 955 | 30. lbtot [..BS]: total number of times a server was selected, either for new |
| 956 | sessions, or when re-dispatching. The server counter is the number |
| 957 | of times that server was selected. |
| 958 | 31. tracked [...S]: id of proxy/server if tracking is enabled. |
| 959 | 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener) |
| 960 | 33. rate [.FBS]: number of sessions per second over last elapsed second |
| 961 | 34. rate_lim [.F..]: configured limit on new sessions per second |
| 962 | 35. rate_max [.FBS]: max number of new sessions per second |
| 963 | 36. check_status [...S]: status of last health check, one of: |
| 964 | UNK -> unknown |
| 965 | INI -> initializing |
| 966 | SOCKERR -> socket error |
| 967 | L4OK -> check passed on layer 4, no upper layers testing enabled |
| 968 | L4TOUT -> layer 1-4 timeout |
| 969 | L4CON -> layer 1-4 connection problem, for example |
| 970 | "Connection refused" (tcp rst) or "No route to host" (icmp) |
| 971 | L6OK -> check passed on layer 6 |
| 972 | L6TOUT -> layer 6 (SSL) timeout |
| 973 | L6RSP -> layer 6 invalid response - protocol error |
| 974 | L7OK -> check passed on layer 7 |
| 975 | L7OKC -> check conditionally passed on layer 7, for example 404 with |
| 976 | disable-on-404 |
| 977 | L7TOUT -> layer 7 (HTTP/SMTP) timeout |
| 978 | L7RSP -> layer 7 invalid response - protocol error |
| 979 | L7STS -> layer 7 response error, for example HTTP 5xx |
| 980 | 37. check_code [...S]: layer5-7 code, if available |
| 981 | 38. check_duration [...S]: time in ms took to finish last health check |
| 982 | 39. hrsp_1xx [.FBS]: http responses with 1xx code |
| 983 | 40. hrsp_2xx [.FBS]: http responses with 2xx code |
| 984 | 41. hrsp_3xx [.FBS]: http responses with 3xx code |
| 985 | 42. hrsp_4xx [.FBS]: http responses with 4xx code |
| 986 | 43. hrsp_5xx [.FBS]: http responses with 5xx code |
| 987 | 44. hrsp_other [.FBS]: http responses with other codes (protocol error) |
| 988 | 45. hanafail [...S]: failed health checks details |
| 989 | 46. req_rate [.F..]: HTTP requests per second over last elapsed second |
| 990 | 47. req_rate_max [.F..]: max number of HTTP requests per second observed |
| 991 | 48. req_tot [.F..]: total number of HTTP requests received |
| 992 | 49. cli_abrt [..BS]: number of data transfers aborted by the client |
| 993 | 50. srv_abrt [..BS]: number of data transfers aborted by the server |
| 994 | (inc. in eresp) |
| 995 | 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor |
| 996 | 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor |
| 997 | 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor |
| 998 | (CPU/BW limit) |
| 999 | 54. comp_rsp [.FB.]: number of HTTP responses that were compressed |
| 1000 | 55. lastsess [..BS]: number of seconds since last session assigned to |
| 1001 | server/backend |
| 1002 | 56. last_chk [...S]: last health check contents or textual error |
| 1003 | 57. last_agt [...S]: last agent check contents or textual error |
| 1004 | 58. qtime [..BS]: the average queue time in ms over the 1024 last requests |
| 1005 | 59. ctime [..BS]: the average connect time in ms over the 1024 last requests |
| 1006 | 60. rtime [..BS]: the average response time in ms over the 1024 last requests |
| 1007 | (0 for TCP) |
| 1008 | 61. ttime [..BS]: the average total session time in ms over the 1024 last |
| 1009 | requests |
Willy Tarreau | 7f61884 | 2016-01-08 11:40:03 +0100 | [diff] [blame] | 1010 | 62. agent_status [...S]: status of last agent check, one of: |
| 1011 | UNK -> unknown |
| 1012 | INI -> initializing |
| 1013 | SOCKERR -> socket error |
| 1014 | L4OK -> check passed on layer 4, no upper layers testing enabled |
| 1015 | L4TOUT -> layer 1-4 timeout |
| 1016 | L4CON -> layer 1-4 connection problem, for example |
| 1017 | "Connection refused" (tcp rst) or "No route to host" (icmp) |
| 1018 | L7OK -> agent reported "up" |
| 1019 | L7STS -> agent reported "fail", "stop", or "down" |
| 1020 | 63. agent_code [...S]: numeric code reported by agent if any (unused for now) |
| 1021 | 64. agent_duration [...S]: time in ms taken to finish last check |
Willy Tarreau | dd7354b | 2016-01-08 13:47:26 +0100 | [diff] [blame] | 1022 | 65. check_desc [...S]: short human-readable description of check_status |
| 1023 | 66. agent_desc [...S]: short human-readable description of agent_status |
Willy Tarreau | 3141f59 | 2016-01-08 14:25:28 +0100 | [diff] [blame] | 1024 | 67. check_rise [...S]: server's "rise" parameter used by checks |
| 1025 | 68. check_fall [...S]: server's "fall" parameter used by checks |
| 1026 | 69. check_health [...S]: server's health check value between 0 and rise+fall-1 |
| 1027 | 70. agent_rise [...S]: agent's "rise" parameter, normally 1 |
| 1028 | 71. agent_fall [...S]: agent's "fall" parameter, normally 1 |
| 1029 | 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1 |
Willy Tarreau | a6f5a73 | 2016-01-08 16:59:56 +0100 | [diff] [blame] | 1030 | 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address. |
Willy Tarreau | e4847c6 | 2016-01-08 15:43:54 +0100 | [diff] [blame] | 1031 | 74: cookie [..BS]: server's cookie value or backend's cookie name |
Willy Tarreau | f8211df | 2016-01-11 14:09:38 +0100 | [diff] [blame] | 1032 | 75: mode [LFBS]: proxy mode (tcp, http, health, unknown) |
Willy Tarreau | f1516d9 | 2016-01-11 14:48:36 +0100 | [diff] [blame] | 1033 | 76: algo [..B.]: load balancing algorithm |
Willy Tarreau | c73810f | 2016-01-11 13:52:04 +0100 | [diff] [blame] | 1034 | 77: conn_rate [.F..]: number of connections over the last elapsed second |
| 1035 | 78: conn_rate_max [.F..]: highest known conn_rate |
| 1036 | 79: conn_tot [.F..]: cumulative number of connections |
Willy Tarreau | 5b9bdff | 2016-01-11 14:40:47 +0100 | [diff] [blame] | 1037 | 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats) |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1038 | |
| 1039 | |
Willy Tarreau | 5d8b979 | 2016-03-11 11:09:34 +0100 | [diff] [blame] | 1040 | 9.2) Typed output format |
| 1041 | ------------------------ |
| 1042 | |
| 1043 | Both "show info" and "show stat" support a mode where each output value comes |
| 1044 | with its type and sufficient information to know how the value is supposed to |
| 1045 | be aggregated between processes and how it evolves. |
| 1046 | |
| 1047 | In all cases, the output consists in having a single value per line with all |
| 1048 | the information split into fields delimited by colons (':'). |
| 1049 | |
| 1050 | The first column designates the object or metric being dumped. Its format is |
| 1051 | specific to the command producing this output and will not be described in this |
| 1052 | section. Usually it will consist in a series of identifiers and field names. |
| 1053 | |
| 1054 | The second column contains 3 characters respectively indicating the origin, the |
| 1055 | nature and the scope of the value being reported. The first character (the |
| 1056 | origin) indicates where the value was extracted from. Possible characters are : |
| 1057 | |
| 1058 | M The value is a metric. It is valid at one instant any may change depending |
| 1059 | on its nature . |
| 1060 | |
| 1061 | S The value is a status. It represents a discrete value which by definition |
| 1062 | cannot be aggregated. It may be the status of a server ("UP" or "DOWN"), |
| 1063 | the PID of the process, etc. |
| 1064 | |
| 1065 | K The value is a sorting key. It represents an identifier which may be used |
| 1066 | to group some values together because it is unique among its class. All |
| 1067 | internal identifiers are keys. Some names can be listed as keys if they |
| 1068 | are unique (eg: a frontend name is unique). In general keys come from the |
| 1069 | configuration, eventhough some of them may automatically be assigned. For |
| 1070 | most purposes keys may be considered as equivalent to configuration. |
| 1071 | |
| 1072 | C The value comes from the configuration. Certain configuration values make |
| 1073 | sense on the output, for example a concurrent connection limit or a cookie |
| 1074 | name. By definition these values are the same in all processes started |
| 1075 | from the same configuration file. |
| 1076 | |
| 1077 | P The value comes from the product itself. There are very few such values, |
| 1078 | most common use is to report the product name, version and release date. |
| 1079 | These elements are also the same between all processes. |
| 1080 | |
| 1081 | The second character (the nature) indicates the nature of the information |
| 1082 | carried by the field in order to let an aggregator decide on what operation to |
| 1083 | use to aggregate multiple values. Possible characters are : |
| 1084 | |
| 1085 | A The value represents an age since a last event. This is a bit different |
| 1086 | from the duration in that an age is automatically computed based on the |
| 1087 | current date. A typical example is how long ago did the last session |
| 1088 | happen on a server. Ages are generally aggregated by taking the minimum |
| 1089 | value and do not need to be stored. |
| 1090 | |
| 1091 | a The value represents an already averaged value. The average response times |
| 1092 | and server weights are of this nature. Averages can typically be averaged |
| 1093 | between processes. |
| 1094 | |
| 1095 | C The value represents a cumulative counter. Such measures perpetually |
| 1096 | increase until they wrap around. Some monitoring protocols need to tell |
| 1097 | the difference between a counter and a gauge to report a different type. |
| 1098 | In general counters may simply be summed since they represent events or |
| 1099 | volumes. Examples of metrics of this nature are connection counts or byte |
| 1100 | counts. |
| 1101 | |
| 1102 | D The value represents a duration for a status. There are a few usages of |
| 1103 | this, most of them include the time taken by the last health check and |
| 1104 | the time a server has spent down. Durations are generally not summed, |
| 1105 | most of the time the maximum will be retained to compute an SLA. |
| 1106 | |
| 1107 | G The value represents a gauge. It's a measure at one instant. The memory |
| 1108 | usage or the current number of active connections are of this nature. |
| 1109 | Metrics of this type are typically summed during aggregation. |
| 1110 | |
| 1111 | L The value represents a limit (generally a configured one). By nature, |
| 1112 | limits are harder to aggregate since they are specific to the point where |
| 1113 | they were retrieved. In certain situations they may be summed or be kept |
| 1114 | separate. |
| 1115 | |
| 1116 | M The value represents a maximum. In general it will apply to a gauge and |
| 1117 | keep the highest known value. An example of such a metric could be the |
| 1118 | maximum amount of concurrent connections that was encountered in the |
| 1119 | product's life time. To correctly aggregate maxima, you are supposed to |
| 1120 | output a range going from the maximum of all maxima and the sum of all |
| 1121 | of them. There is indeed no way to know if they were encountered |
| 1122 | simultaneously or not. |
| 1123 | |
| 1124 | m The value represents a minimum. In general it will apply to a gauge and |
| 1125 | keep the lowest known value. An example of such a metric could be the |
| 1126 | minimum amount of free memory pools that was encountered in the product's |
| 1127 | life time. To correctly aggregate minima, you are supposed to output a |
| 1128 | range going from the minimum of all minima and the sum of all of them. |
| 1129 | There is indeed no way to know if they were encountered simultaneously |
| 1130 | or not. |
| 1131 | |
| 1132 | N The value represents a name, so it is a string. It is used to report |
| 1133 | proxy names, server names and cookie names. Names have configuration or |
| 1134 | keys as their origin and are supposed to be the same among all processes. |
| 1135 | |
| 1136 | O The value represents a free text output. Outputs from various commands, |
| 1137 | returns from health checks, node descriptions are of such nature. |
| 1138 | |
| 1139 | R The value represents an event rate. It's a measure at one instant. It is |
| 1140 | quite similar to a gauge except that the recipient knows that this measure |
| 1141 | moves slowly and may decide not to keep all values. An example of such a |
| 1142 | metric is the measured amount of connections per second. Metrics of this |
| 1143 | type are typically summed during aggregation. |
| 1144 | |
| 1145 | T The value represents a date or time. A field emitting the current date |
| 1146 | would be of this type. The method to aggregate such information is left |
| 1147 | as an implementation choice. For now no field uses this type. |
| 1148 | |
| 1149 | The third character (the scope) indicates what extent the value reflects. Some |
| 1150 | elements may be per process while others may be per configuration or per system. |
| 1151 | The distinction is important to know whether or not a single value should be |
| 1152 | kept during aggregation or if values have to be aggregated. The following |
| 1153 | characters are currently supported : |
| 1154 | |
| 1155 | C The value is valid for a whole cluster of nodes, which is the set of nodes |
| 1156 | communicating over the peers protocol. An example could be the amount of |
| 1157 | entries present in a stick table that is replicated with other peers. At |
| 1158 | the moment no metric use this scope. |
| 1159 | |
| 1160 | P The value is valid only for the process reporting it. Most metrics use |
| 1161 | this scope. |
| 1162 | |
| 1163 | S The value is valid for the whole service, which is the set of processes |
| 1164 | started together from the same configuration file. All metrics originating |
| 1165 | from the configuration use this scope. Some other metrics may use it as |
| 1166 | well for some shared resources (eg: shared SSL cache statistics). |
| 1167 | |
| 1168 | s The value is valid for the whole system, such as the system's hostname, |
| 1169 | current date or resource usage. At the moment this scope is not used by |
| 1170 | any metric. |
| 1171 | |
| 1172 | Consumers of these information will generally have enough of these 3 characters |
| 1173 | to determine how to accurately report aggregated information across multiple |
| 1174 | processes. |
| 1175 | |
| 1176 | After this column, the third column indicates the type of the field, among "s32" |
| 1177 | (signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit |
| 1178 | integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to |
| 1179 | know the type before parsing the value in order to properly read it. For example |
| 1180 | a string containing only digits is still a string an not an integer (eg: an |
| 1181 | error code extracted by a check). |
| 1182 | |
| 1183 | Then the fourth column is the value itself, encoded according to its type. |
| 1184 | Strings are dumped as-is immediately after the colon without any leading space. |
| 1185 | If a string contains a colon, it will appear normally. This means that the |
| 1186 | output should not be exclusively split around colons or some check outputs |
| 1187 | or server addresses might be truncated. |
| 1188 | |
| 1189 | |
| 1190 | 9.3. Unix Socket commands |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1191 | ------------------------- |
| 1192 | |
| 1193 | The stats socket is not enabled by default. In order to enable it, it is |
| 1194 | necessary to add one line in the global section of the haproxy configuration. |
| 1195 | A second line is recommended to set a larger timeout, always appreciated when |
| 1196 | issuing commands by hand : |
| 1197 | |
| 1198 | global |
| 1199 | stats socket /var/run/haproxy.sock mode 600 level admin |
| 1200 | stats timeout 2m |
| 1201 | |
| 1202 | It is also possible to add multiple instances of the stats socket by repeating |
| 1203 | the line, and make them listen to a TCP port instead of a UNIX socket. This is |
| 1204 | never done by default because this is dangerous, but can be handy in some |
| 1205 | situations : |
| 1206 | |
| 1207 | global |
| 1208 | stats socket /var/run/haproxy.sock mode 600 level admin |
| 1209 | stats socket ipv4@192.168.0.1:9999 level admin |
| 1210 | stats timeout 2m |
| 1211 | |
| 1212 | To access the socket, an external utility such as "socat" is required. Socat is |
| 1213 | a swiss-army knife to connect anything to anything. We use it to connect |
| 1214 | terminals to the socket, or a couple of stdin/stdout pipes to it for scripts. |
| 1215 | The two main syntaxes we'll use are the following : |
| 1216 | |
| 1217 | # socat /var/run/haproxy.sock stdio |
| 1218 | # socat /var/run/haproxy.sock readline |
| 1219 | |
| 1220 | The first one is used with scripts. It is possible to send the output of a |
| 1221 | script to haproxy, and pass haproxy's output to another script. That's useful |
| 1222 | for retrieving counters or attack traces for example. |
| 1223 | |
| 1224 | The second one is only useful for issuing commands by hand. It has the benefit |
| 1225 | that the terminal is handled by the readline library which supports line |
| 1226 | editing and history, which is very convenient when issuing repeated commands |
| 1227 | (eg: watch a counter). |
| 1228 | |
| 1229 | The socket supports two operation modes : |
| 1230 | - interactive |
| 1231 | - non-interactive |
| 1232 | |
| 1233 | The non-interactive mode is the default when socat connects to the socket. In |
| 1234 | this mode, a single line may be sent. It is processed as a whole, responses are |
| 1235 | sent back, and the connection closes after the end of the response. This is the |
| 1236 | mode that scripts and monitoring tools use. It is possible to send multiple |
| 1237 | commands in this mode, they need to be delimited by a semi-colon (';'). For |
| 1238 | example : |
| 1239 | |
| 1240 | # echo "show info;show stat;show table" | socat /var/run/haproxy stdio |
| 1241 | |
| 1242 | The interactive mode displays a prompt ('>') and waits for commands to be |
| 1243 | entered on the line, then processes them, and displays the prompt again to wait |
| 1244 | for a new command. This mode is entered via the "prompt" command which must be |
| 1245 | sent on the first line in non-interactive mode. The mode is a flip switch, if |
| 1246 | "prompt" is sent in interactive mode, it is disabled and the connection closes |
| 1247 | after processing the last command of the same line. |
| 1248 | |
| 1249 | For this reason, when debugging by hand, it's quite common to start with the |
| 1250 | "prompt" command : |
| 1251 | |
| 1252 | # socat /var/run/haproxy readline |
| 1253 | prompt |
| 1254 | > show info |
| 1255 | ... |
| 1256 | > |
| 1257 | |
| 1258 | Since multiple commands may be issued at once, haproxy uses the empty line as a |
| 1259 | delimiter to mark an end of output for each command, and takes care of ensuring |
| 1260 | that no command can emit an empty line on output. A script can thus easily |
| 1261 | parse the output even when multiple commands were pipelined on a single line. |
| 1262 | |
| 1263 | It is important to understand that when multiple haproxy processes are started |
| 1264 | on the same sockets, any process may pick up the request and will output its |
| 1265 | own stats. |
| 1266 | |
| 1267 | The list of commands currently supported on the stats socket is provided below. |
| 1268 | If an unknown command is sent, haproxy displays the usage message which reminds |
| 1269 | all supported commands. Some commands support a more complex syntax, generally |
| 1270 | it will explain what part of the command is invalid when this happens. |
| 1271 | |
| 1272 | add acl <acl> <pattern> |
| 1273 | Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by |
| 1274 | "show acl". This command does not verify if the entry already exists. This |
| 1275 | command cannot be used if the reference <acl> is a file also used with a map. |
| 1276 | In this case, you must use the command "add map" in place of "add acl". |
| 1277 | |
| 1278 | add map <map> <key> <value> |
| 1279 | Add an entry into the map <map> to associate the value <value> to the key |
| 1280 | <key>. This command does not verify if the entry already exists. It is |
| 1281 | mainly used to fill a map after a clear operation. Note that if the reference |
| 1282 | <map> is a file and is shared with a map, this map will contain also a new |
| 1283 | pattern entry. |
| 1284 | |
| 1285 | clear counters |
| 1286 | Clear the max values of the statistics counters in each proxy (frontend & |
| 1287 | backend) and in each server. The cumulated counters are not affected. This |
| 1288 | can be used to get clean counters after an incident, without having to |
| 1289 | restart nor to clear traffic counters. This command is restricted and can |
| 1290 | only be issued on sockets configured for levels "operator" or "admin". |
| 1291 | |
| 1292 | clear counters all |
| 1293 | Clear all statistics counters in each proxy (frontend & backend) and in each |
| 1294 | server. This has the same effect as restarting. This command is restricted |
| 1295 | and can only be issued on sockets configured for level "admin". |
| 1296 | |
| 1297 | clear acl <acl> |
| 1298 | Remove all entries from the acl <acl>. <acl> is the #<id> or the <file> |
| 1299 | returned by "show acl". Note that if the reference <acl> is a file and is |
| 1300 | shared with a map, this map will be also cleared. |
| 1301 | |
| 1302 | clear map <map> |
| 1303 | Remove all entries from the map <map>. <map> is the #<id> or the <file> |
| 1304 | returned by "show map". Note that if the reference <map> is a file and is |
| 1305 | shared with a acl, this acl will be also cleared. |
| 1306 | |
| 1307 | clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ] |
| 1308 | Remove entries from the stick-table <table>. |
| 1309 | |
| 1310 | This is typically used to unblock some users complaining they have been |
| 1311 | abusively denied access to a service, but this can also be used to clear some |
| 1312 | stickiness entries matching a server that is going to be replaced (see "show |
| 1313 | table" below for details). Note that sometimes, removal of an entry will be |
| 1314 | refused because it is currently tracked by a session. Retrying a few seconds |
| 1315 | later after the session ends is usual enough. |
| 1316 | |
| 1317 | In the case where no options arguments are given all entries will be removed. |
| 1318 | |
| 1319 | When the "data." form is used entries matching a filter applied using the |
| 1320 | stored data (see "stick-table" in section 4.2) are removed. A stored data |
| 1321 | type must be specified in <type>, and this data type must be stored in the |
| 1322 | table otherwise an error is reported. The data is compared according to |
| 1323 | <operator> with the 64-bit integer <value>. Operators are the same as with |
| 1324 | the ACLs : |
| 1325 | |
| 1326 | - eq : match entries whose data is equal to this value |
| 1327 | - ne : match entries whose data is not equal to this value |
| 1328 | - le : match entries whose data is less than or equal to this value |
| 1329 | - ge : match entries whose data is greater than or equal to this value |
| 1330 | - lt : match entries whose data is less than this value |
| 1331 | - gt : match entries whose data is greater than this value |
| 1332 | |
| 1333 | When the key form is used the entry <key> is removed. The key must be of the |
| 1334 | same type as the table, which currently is limited to IPv4, IPv6, integer and |
| 1335 | string. |
| 1336 | |
| 1337 | Example : |
| 1338 | $ echo "show table http_proxy" | socat stdio /tmp/sock1 |
| 1339 | >>> # table: http_proxy, type: ip, size:204800, used:2 |
| 1340 | >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \ |
| 1341 | bytes_out_rate(60000)=187 |
| 1342 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 1343 | bytes_out_rate(60000)=191 |
| 1344 | |
| 1345 | $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1 |
| 1346 | |
| 1347 | $ echo "show table http_proxy" | socat stdio /tmp/sock1 |
| 1348 | >>> # table: http_proxy, type: ip, size:204800, used:1 |
| 1349 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 1350 | bytes_out_rate(60000)=191 |
| 1351 | $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1 |
| 1352 | $ echo "show table http_proxy" | socat stdio /tmp/sock1 |
| 1353 | >>> # table: http_proxy, type: ip, size:204800, used:1 |
| 1354 | |
| 1355 | del acl <acl> [<key>|#<ref>] |
| 1356 | Delete all the acl entries from the acl <acl> corresponding to the key <key>. |
| 1357 | <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used, |
| 1358 | this command delete only the listed reference. The reference can be found with |
| 1359 | listing the content of the acl. Note that if the reference <acl> is a file and |
| 1360 | is shared with a map, the entry will be also deleted in the map. |
| 1361 | |
| 1362 | del map <map> [<key>|#<ref>] |
| 1363 | Delete all the map entries from the map <map> corresponding to the key <key>. |
| 1364 | <map> is the #<id> or the <file> returned by "show map". If the <ref> is used, |
| 1365 | this command delete only the listed reference. The reference can be found with |
| 1366 | listing the content of the map. Note that if the reference <map> is a file and |
| 1367 | is shared with a acl, the entry will be also deleted in the map. |
| 1368 | |
| 1369 | disable agent <backend>/<server> |
| 1370 | Mark the auxiliary agent check as temporarily stopped. |
| 1371 | |
| 1372 | In the case where an agent check is being run as a auxiliary check, due |
| 1373 | to the agent-check parameter of a server directive, new checks are only |
| 1374 | initialised when the agent is in the enabled. Thus, disable agent will |
| 1375 | prevent any new agent checks from begin initiated until the agent |
| 1376 | re-enabled using enable agent. |
| 1377 | |
| 1378 | When an agent is disabled the processing of an auxiliary agent check that |
| 1379 | was initiated while the agent was set as enabled is as follows: All |
| 1380 | results that would alter the weight, specifically "drain" or a weight |
| 1381 | returned by the agent, are ignored. The processing of agent check is |
| 1382 | otherwise unchanged. |
| 1383 | |
| 1384 | The motivation for this feature is to allow the weight changing effects |
| 1385 | of the agent checks to be paused to allow the weight of a server to be |
| 1386 | configured using set weight without being overridden by the agent. |
| 1387 | |
| 1388 | This command is restricted and can only be issued on sockets configured for |
| 1389 | level "admin". |
| 1390 | |
| 1391 | disable frontend <frontend> |
| 1392 | Mark the frontend as temporarily stopped. This corresponds to the mode which |
| 1393 | is used during a soft restart : the frontend releases the port but can be |
| 1394 | enabled again if needed. This should be used with care as some non-Linux OSes |
| 1395 | are unable to enable it back. This is intended to be used in environments |
| 1396 | where stopping a proxy is not even imaginable but a misconfigured proxy must |
| 1397 | be fixed. That way it's possible to release the port and bind it into another |
| 1398 | process to restore operations. The frontend will appear with status "STOP" |
| 1399 | on the stats page. |
| 1400 | |
| 1401 | The frontend may be specified either by its name or by its numeric ID, |
| 1402 | prefixed with a sharp ('#'). |
| 1403 | |
| 1404 | This command is restricted and can only be issued on sockets configured for |
| 1405 | level "admin". |
| 1406 | |
| 1407 | disable health <backend>/<server> |
| 1408 | Mark the primary health check as temporarily stopped. This will disable |
| 1409 | sending of health checks, and the last health check result will be ignored. |
| 1410 | The server will be in unchecked state and considered UP unless an auxiliary |
| 1411 | agent check forces it down. |
| 1412 | |
| 1413 | This command is restricted and can only be issued on sockets configured for |
| 1414 | level "admin". |
| 1415 | |
| 1416 | disable server <backend>/<server> |
| 1417 | Mark the server DOWN for maintenance. In this mode, no more checks will be |
| 1418 | performed on the server until it leaves maintenance. |
| 1419 | If the server is tracked by other servers, those servers will be set to DOWN |
| 1420 | during the maintenance. |
| 1421 | |
| 1422 | In the statistics page, a server DOWN for maintenance will appear with a |
| 1423 | "MAINT" status, its tracking servers with the "MAINT(via)" one. |
| 1424 | |
| 1425 | Both the backend and the server may be specified either by their name or by |
| 1426 | their numeric ID, prefixed with a sharp ('#'). |
| 1427 | |
| 1428 | This command is restricted and can only be issued on sockets configured for |
| 1429 | level "admin". |
| 1430 | |
| 1431 | enable agent <backend>/<server> |
| 1432 | Resume auxiliary agent check that was temporarily stopped. |
| 1433 | |
| 1434 | See "disable agent" for details of the effect of temporarily starting |
| 1435 | and stopping an auxiliary agent. |
| 1436 | |
| 1437 | This command is restricted and can only be issued on sockets configured for |
| 1438 | level "admin". |
| 1439 | |
| 1440 | enable frontend <frontend> |
| 1441 | Resume a frontend which was temporarily stopped. It is possible that some of |
| 1442 | the listening ports won't be able to bind anymore (eg: if another process |
| 1443 | took them since the 'disable frontend' operation). If this happens, an error |
| 1444 | is displayed. Some operating systems might not be able to resume a frontend |
| 1445 | which was disabled. |
| 1446 | |
| 1447 | The frontend may be specified either by its name or by its numeric ID, |
| 1448 | prefixed with a sharp ('#'). |
| 1449 | |
| 1450 | This command is restricted and can only be issued on sockets configured for |
| 1451 | level "admin". |
| 1452 | |
| 1453 | enable health <backend>/<server> |
| 1454 | Resume a primary health check that was temporarily stopped. This will enable |
| 1455 | sending of health checks again. Please see "disable health" for details. |
| 1456 | |
| 1457 | This command is restricted and can only be issued on sockets configured for |
| 1458 | level "admin". |
| 1459 | |
| 1460 | enable server <backend>/<server> |
| 1461 | If the server was previously marked as DOWN for maintenance, this marks the |
| 1462 | server UP and checks are re-enabled. |
| 1463 | |
| 1464 | Both the backend and the server may be specified either by their name or by |
| 1465 | their numeric ID, prefixed with a sharp ('#'). |
| 1466 | |
| 1467 | This command is restricted and can only be issued on sockets configured for |
| 1468 | level "admin". |
| 1469 | |
| 1470 | get map <map> <value> |
| 1471 | get acl <acl> <value> |
| 1472 | Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl> |
| 1473 | are the #<id> or the <file> returned by "show map" or "show acl". This command |
| 1474 | returns all the matching patterns associated with this map. This is useful for |
| 1475 | debugging maps and ACLs. The output format is composed by one line par |
| 1476 | matching type. Each line is composed by space-delimited series of words. |
| 1477 | |
| 1478 | The first two words are: |
| 1479 | |
| 1480 | <match method>: The match method applied. It can be "found", "bool", |
| 1481 | "int", "ip", "bin", "len", "str", "beg", "sub", "dir", |
| 1482 | "dom", "end" or "reg". |
| 1483 | |
| 1484 | <match result>: The result. Can be "match" or "no-match". |
| 1485 | |
| 1486 | The following words are returned only if the pattern matches an entry. |
| 1487 | |
| 1488 | <index type>: "tree" or "list". The internal lookup algorithm. |
| 1489 | |
| 1490 | <case>: "case-insensitive" or "case-sensitive". The |
| 1491 | interpretation of the case. |
| 1492 | |
| 1493 | <entry matched>: match="<entry>". Return the matched pattern. It is |
| 1494 | useful with regular expressions. |
| 1495 | |
| 1496 | The two last word are used to show the returned value and its type. With the |
| 1497 | "acl" case, the pattern doesn't exist. |
| 1498 | |
| 1499 | return=nothing: No return because there are no "map". |
| 1500 | return="<value>": The value returned in the string format. |
| 1501 | return=cannot-display: The value cannot be converted as string. |
| 1502 | |
| 1503 | type="<type>": The type of the returned sample. |
| 1504 | |
| 1505 | get weight <backend>/<server> |
| 1506 | Report the current weight and the initial weight of server <server> in |
| 1507 | backend <backend> or an error if either doesn't exist. The initial weight is |
| 1508 | the one that appears in the configuration file. Both are normally equal |
| 1509 | unless the current weight has been changed. Both the backend and the server |
| 1510 | may be specified either by their name or by their numeric ID, prefixed with a |
| 1511 | sharp ('#'). |
| 1512 | |
| 1513 | help |
| 1514 | Print the list of known keywords and their basic usage. The same help screen |
| 1515 | is also displayed for unknown commands. |
| 1516 | |
| 1517 | prompt |
| 1518 | Toggle the prompt at the beginning of the line and enter or leave interactive |
| 1519 | mode. In interactive mode, the connection is not closed after a command |
| 1520 | completes. Instead, the prompt will appear again, indicating the user that |
| 1521 | the interpreter is waiting for a new command. The prompt consists in a right |
| 1522 | angle bracket followed by a space "> ". This mode is particularly convenient |
| 1523 | when one wants to periodically check information such as stats or errors. |
| 1524 | It is also a good idea to enter interactive mode before issuing a "help" |
| 1525 | command. |
| 1526 | |
| 1527 | quit |
| 1528 | Close the connection when in interactive mode. |
| 1529 | |
| 1530 | set map <map> [<key>|#<ref>] <value> |
| 1531 | Modify the value corresponding to each key <key> in a map <map>. <map> is the |
| 1532 | #<id> or <file> returned by "show map". If the <ref> is used in place of |
| 1533 | <key>, only the entry pointed by <ref> is changed. The new value is <value>. |
| 1534 | |
| 1535 | set maxconn frontend <frontend> <value> |
| 1536 | Dynamically change the specified frontend's maxconn setting. Any positive |
| 1537 | value is allowed including zero, but setting values larger than the global |
| 1538 | maxconn does not make much sense. If the limit is increased and connections |
| 1539 | were pending, they will immediately be accepted. If it is lowered to a value |
| 1540 | below the current number of connections, new connections acceptation will be |
| 1541 | delayed until the threshold is reached. The frontend might be specified by |
| 1542 | either its name or its numeric ID prefixed with a sharp ('#'). |
| 1543 | |
Andrew Hayworth | edb93a7 | 2015-10-27 21:46:25 +0000 | [diff] [blame] | 1544 | set maxconn server <backend/server> <value> |
| 1545 | Dynamically change the specified server's maxconn setting. Any positive |
| 1546 | value is allowed including zero, but setting values larger than the global |
| 1547 | maxconn does not make much sense. |
| 1548 | |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1549 | set maxconn global <maxconn> |
| 1550 | Dynamically change the global maxconn setting within the range defined by the |
| 1551 | initial global maxconn setting. If it is increased and connections were |
| 1552 | pending, they will immediately be accepted. If it is lowered to a value below |
| 1553 | the current number of connections, new connections acceptation will be |
| 1554 | delayed until the threshold is reached. A value of zero restores the initial |
| 1555 | setting. |
| 1556 | |
| 1557 | set rate-limit connections global <value> |
| 1558 | Change the process-wide connection rate limit, which is set by the global |
| 1559 | 'maxconnrate' setting. A value of zero disables the limitation. This limit |
| 1560 | applies to all frontends and the change has an immediate effect. The value |
| 1561 | is passed in number of connections per second. |
| 1562 | |
| 1563 | set rate-limit http-compression global <value> |
| 1564 | Change the maximum input compression rate, which is set by the global |
| 1565 | 'maxcomprate' setting. A value of zero disables the limitation. The value is |
| 1566 | passed in number of kilobytes per second. The value is available in the "show |
| 1567 | info" on the line "CompressBpsRateLim" in bytes. |
| 1568 | |
| 1569 | set rate-limit sessions global <value> |
| 1570 | Change the process-wide session rate limit, which is set by the global |
| 1571 | 'maxsessrate' setting. A value of zero disables the limitation. This limit |
| 1572 | applies to all frontends and the change has an immediate effect. The value |
| 1573 | is passed in number of sessions per second. |
| 1574 | |
| 1575 | set rate-limit ssl-sessions global <value> |
| 1576 | Change the process-wide SSL session rate limit, which is set by the global |
| 1577 | 'maxsslrate' setting. A value of zero disables the limitation. This limit |
| 1578 | applies to all frontends and the change has an immediate effect. The value |
| 1579 | is passed in number of sessions per second sent to the SSL stack. It applies |
| 1580 | before the handshake in order to protect the stack against handshake abuses. |
| 1581 | |
| 1582 | set server <backend>/<server> addr <ip4 or ip6 address> |
| 1583 | Replace the current IP address of a server by the one provided. |
| 1584 | |
| 1585 | set server <backend>/<server> agent [ up | down ] |
| 1586 | Force a server's agent to a new state. This can be useful to immediately |
| 1587 | switch a server's state regardless of some slow agent checks for example. |
| 1588 | Note that the change is propagated to tracking servers if any. |
| 1589 | |
| 1590 | set server <backend>/<server> health [ up | stopping | down ] |
| 1591 | Force a server's health to a new state. This can be useful to immediately |
| 1592 | switch a server's state regardless of some slow health checks for example. |
| 1593 | Note that the change is propagated to tracking servers if any. |
| 1594 | |
| 1595 | set server <backend>/<server> state [ ready | drain | maint ] |
| 1596 | Force a server's administrative state to a new state. This can be useful to |
| 1597 | disable load balancing and/or any traffic to a server. Setting the state to |
| 1598 | "ready" puts the server in normal mode, and the command is the equivalent of |
| 1599 | the "enable server" command. Setting the state to "maint" disables any traffic |
| 1600 | to the server as well as any health checks. This is the equivalent of the |
| 1601 | "disable server" command. Setting the mode to "drain" only removes the server |
| 1602 | from load balancing but still allows it to be checked and to accept new |
| 1603 | persistent connections. Changes are propagated to tracking servers if any. |
| 1604 | |
| 1605 | set server <backend>/<server> weight <weight>[%] |
| 1606 | Change a server's weight to the value passed in argument. This is the exact |
| 1607 | equivalent of the "set weight" command below. |
| 1608 | |
| 1609 | set ssl ocsp-response <response> |
| 1610 | This command is used to update an OCSP Response for a certificate (see "crt" |
| 1611 | on "bind" lines). Same controls are performed as during the initial loading of |
| 1612 | the response. The <response> must be passed as a base64 encoded string of the |
| 1613 | DER encoded response from the OCSP server. |
| 1614 | |
| 1615 | Example: |
| 1616 | openssl ocsp -issuer issuer.pem -cert server.pem \ |
| 1617 | -host ocsp.issuer.com:80 -respout resp.der |
| 1618 | echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \ |
| 1619 | socat stdio /var/run/haproxy.stat |
| 1620 | |
| 1621 | set ssl tls-key <id> <tlskey> |
| 1622 | Set the next TLS key for the <id> listener to <tlskey>. This key becomes the |
| 1623 | ultimate key, while the penultimate one is used for encryption (others just |
| 1624 | decrypt). The oldest TLS key present is overwritten. <id> is either a numeric |
| 1625 | #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48 |
| 1626 | bit TLS ticket key (ex. openssl rand -base64 48). |
| 1627 | |
| 1628 | set table <table> key <key> [data.<data_type> <value>]* |
| 1629 | Create or update a stick-table entry in the table. If the key is not present, |
| 1630 | an entry is inserted. See stick-table in section 4.2 to find all possible |
| 1631 | values for <data_type>. The most likely use consists in dynamically entering |
| 1632 | entries for source IP addresses, with a flag in gpc0 to dynamically block an |
| 1633 | IP address or affect its quality of service. It is possible to pass multiple |
| 1634 | data_types in a single call. |
| 1635 | |
| 1636 | set timeout cli <delay> |
| 1637 | Change the CLI interface timeout for current connection. This can be useful |
| 1638 | during long debugging sessions where the user needs to constantly inspect |
| 1639 | some indicators without being disconnected. The delay is passed in seconds. |
| 1640 | |
| 1641 | set weight <backend>/<server> <weight>[%] |
| 1642 | Change a server's weight to the value passed in argument. If the value ends |
| 1643 | with the '%' sign, then the new weight will be relative to the initially |
| 1644 | configured weight. Absolute weights are permitted between 0 and 256. |
| 1645 | Relative weights must be positive with the resulting absolute weight is |
| 1646 | capped at 256. Servers which are part of a farm running a static |
| 1647 | load-balancing algorithm have stricter limitations because the weight |
| 1648 | cannot change once set. Thus for these servers, the only accepted values |
| 1649 | are 0 and 100% (or 0 and the initial weight). Changes take effect |
| 1650 | immediately, though certain LB algorithms require a certain amount of |
| 1651 | requests to consider changes. A typical usage of this command is to |
| 1652 | disable a server during an update by setting its weight to zero, then to |
| 1653 | enable it again after the update by setting it back to 100%. This command |
| 1654 | is restricted and can only be issued on sockets configured for level |
| 1655 | "admin". Both the backend and the server may be specified either by their |
| 1656 | name or by their numeric ID, prefixed with a sharp ('#'). |
| 1657 | |
Willy Tarreau | ae79572 | 2016-02-16 11:27:28 +0100 | [diff] [blame] | 1658 | show env [<name>] |
| 1659 | Dump one or all environment variables known by the process. Without any |
| 1660 | argument, all variables are dumped. With an argument, only the specified |
| 1661 | variable is dumped if it exists. Otherwise "Variable not found" is emitted. |
| 1662 | Variables are dumped in the same format as they are stored or returned by the |
| 1663 | "env" utility, that is, "<name>=<value>". This can be handy when debugging |
| 1664 | certain configuration files making heavy use of environment variables to |
| 1665 | ensure that they contain the expected values. This command is restricted and |
| 1666 | can only be issued on sockets configured for levels "operator" or "admin". |
| 1667 | |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1668 | show errors [<iid>] |
| 1669 | Dump last known request and response errors collected by frontends and |
| 1670 | backends. If <iid> is specified, the limit the dump to errors concerning |
| 1671 | either frontend or backend whose ID is <iid>. This command is restricted |
| 1672 | and can only be issued on sockets configured for levels "operator" or |
| 1673 | "admin". |
| 1674 | |
| 1675 | The errors which may be collected are the last request and response errors |
| 1676 | caused by protocol violations, often due to invalid characters in header |
| 1677 | names. The report precisely indicates what exact character violated the |
| 1678 | protocol. Other important information such as the exact date the error was |
| 1679 | detected, frontend and backend names, the server name (when known), the |
| 1680 | internal session ID and the source address which has initiated the session |
| 1681 | are reported too. |
| 1682 | |
| 1683 | All characters are returned, and non-printable characters are encoded. The |
| 1684 | most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one |
| 1685 | letter following a backslash. The backslash itself is encoded as '\\' to |
| 1686 | avoid confusion. Other non-printable characters are encoded '\xNN' where |
| 1687 | NN is the two-digits hexadecimal representation of the character's ASCII |
| 1688 | code. |
| 1689 | |
| 1690 | Lines are prefixed with the position of their first character, starting at 0 |
| 1691 | for the beginning of the buffer. At most one input line is printed per line, |
| 1692 | and large lines will be broken into multiple consecutive output lines so that |
| 1693 | the output never goes beyond 79 characters wide. It is easy to detect if a |
| 1694 | line was broken, because it will not end with '\n' and the next line's offset |
| 1695 | will be followed by a '+' sign, indicating it is a continuation of previous |
| 1696 | line. |
| 1697 | |
| 1698 | Example : |
| 1699 | $ echo "show errors" | socat stdio /tmp/sock1 |
| 1700 | >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response |
| 1701 | src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1) |
| 1702 | response length 213 bytes, error at position 23: |
| 1703 | |
| 1704 | 00000 HTTP/1.0 200 OK\r\n |
| 1705 | 00017 header/bizarre:blah\r\n |
| 1706 | 00038 Location: blah\r\n |
| 1707 | 00054 Long-line: this is a very long line which should b |
| 1708 | 00104+ e broken into multiple lines on the output buffer, |
| 1709 | 00154+ otherwise it would be too large to print in a ter |
| 1710 | 00204+ minal\r\n |
| 1711 | 00211 \r\n |
| 1712 | |
| 1713 | In the example above, we see that the backend "http-in" which has internal |
| 1714 | ID 2 has blocked an invalid response from its server s2 which has internal |
| 1715 | ID 1. The request was on session 54 initiated by source 127.0.0.1 and |
| 1716 | received by frontend fe-eth0 whose ID is 1. The total response length was |
| 1717 | 213 bytes when the error was detected, and the error was at byte 23. This |
| 1718 | is the slash ('/') in header name "header/bizarre", which is not a valid |
| 1719 | HTTP character for a header name. |
| 1720 | |
| 1721 | show backend |
| 1722 | Dump the list of backends available in the running process |
| 1723 | |
Willy Tarreau | 5d8b979 | 2016-03-11 11:09:34 +0100 | [diff] [blame] | 1724 | show info [typed] |
| 1725 | Dump info about haproxy status on current process. If "typed" is passed as an |
| 1726 | optional argument, field numbers, names and types are emitted as well so that |
| 1727 | external monitoring products can easily retrieve, possibly aggregate, then |
| 1728 | report information found in fields they don't know. Each field is dumped on |
| 1729 | its own line. By default, the format contains only two columns delimited by a |
| 1730 | colon (':'). The left one is the field name and the right one is the value. |
| 1731 | It is very important to note that in typed output format, the dump for a |
| 1732 | single object is contigous so that there is no need for a consumer to store |
| 1733 | everything at once. |
| 1734 | |
| 1735 | When using the typed output format, each line is made of 4 columns delimited |
| 1736 | by colons (':'). The first column is a dot-delimited series of 3 elements. The |
| 1737 | first element is the numeric position of the field in the list (starting at |
| 1738 | zero). This position shall not change over time, but holes are to be expected, |
| 1739 | depending on build options or if some fields are deleted in the future. The |
| 1740 | second element is the field name as it appears in the default "show info" |
| 1741 | output. The third element is the relative process number starting at 1. |
| 1742 | |
| 1743 | The rest of the line starting after the first colon follows the "typed output |
| 1744 | format" described in the section above. In short, the second column (after the |
| 1745 | first ':') indicates the origin, nature and scope of the variable. The third |
| 1746 | column indicates the type of the field, among "s32", "s64", "u32", "u64" and |
| 1747 | "str". Then the fourth column is the value itself, which the consumer knows |
| 1748 | how to parse thanks to column 3 and how to process thanks to column 2. |
| 1749 | |
| 1750 | Thus the overall line format in typed mode is : |
| 1751 | |
| 1752 | <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value> |
| 1753 | |
| 1754 | Example : |
| 1755 | |
| 1756 | > show info |
| 1757 | Name: HAProxy |
| 1758 | Version: 1.7-dev1-de52ea-146 |
| 1759 | Release_date: 2016/03/11 |
| 1760 | Nbproc: 1 |
| 1761 | Process_num: 1 |
| 1762 | Pid: 28105 |
| 1763 | Uptime: 0d 0h00m04s |
| 1764 | Uptime_sec: 4 |
| 1765 | Memmax_MB: 0 |
| 1766 | PoolAlloc_MB: 0 |
| 1767 | PoolUsed_MB: 0 |
| 1768 | PoolFailed: 0 |
| 1769 | (...) |
| 1770 | |
| 1771 | > show info typed |
| 1772 | 0.Name.1:POS:str:HAProxy |
| 1773 | 1.Version.1:POS:str:1.7-dev1-de52ea-146 |
| 1774 | 2.Release_date.1:POS:str:2016/03/11 |
| 1775 | 3.Nbproc.1:CGS:u32:1 |
| 1776 | 4.Process_num.1:KGP:u32:1 |
| 1777 | 5.Pid.1:SGP:u32:28105 |
| 1778 | 6.Uptime.1:MDP:str:0d 0h00m08s |
| 1779 | 7.Uptime_sec.1:MDP:u32:8 |
| 1780 | 8.Memmax_MB.1:CLP:u32:0 |
| 1781 | 9.PoolAlloc_MB.1:MGP:u32:0 |
| 1782 | 10.PoolUsed_MB.1:MGP:u32:0 |
| 1783 | 11.PoolFailed.1:MCP:u32:0 |
| 1784 | (...) |
| 1785 | |
| 1786 | In the typed format, the presence of the process ID at the end of the line |
| 1787 | makes it very easy to visually aggregate outputs from multiple processes. |
| 1788 | Example : |
| 1789 | |
| 1790 | $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \ |
| 1791 | echo show info typed | socat /var/run/haproxy.sock2 ) | \ |
| 1792 | sort -t . -k 1,1n -k 2,2 -k 3,3n |
| 1793 | 0.Name.1:POS:str:HAProxy |
| 1794 | 0.Name.2:POS:str:HAProxy |
| 1795 | 1.Version.1:POS:str:1.7-dev1-868ab3-148 |
| 1796 | 1.Version.2:POS:str:1.7-dev1-868ab3-148 |
| 1797 | 2.Release_date.1:POS:str:2016/03/11 |
| 1798 | 2.Release_date.2:POS:str:2016/03/11 |
| 1799 | 3.Nbproc.1:CGS:u32:2 |
| 1800 | 3.Nbproc.2:CGS:u32:2 |
| 1801 | 4.Process_num.1:KGP:u32:1 |
| 1802 | 4.Process_num.2:KGP:u32:2 |
| 1803 | 5.Pid.1:SGP:u32:30120 |
| 1804 | 5.Pid.2:SGP:u32:30121 |
| 1805 | 6.Uptime.1:MDP:str:0d 0h01m28s |
| 1806 | 6.Uptime.2:MDP:str:0d 0h01m28s |
| 1807 | (...) |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1808 | |
| 1809 | show map [<map>] |
| 1810 | Dump info about map converters. Without argument, the list of all available |
| 1811 | maps is returned. If a <map> is specified, its contents are dumped. <map> is |
| 1812 | the #<id> or <file>. The first column is a unique identifier. It can be used |
| 1813 | as reference for the operation "del map" and "set map". The second column is |
| 1814 | the pattern and the third column is the sample if available. The data returned |
| 1815 | are not directly a list of available maps, but are the list of all patterns |
| 1816 | composing any map. Many of these patterns can be shared with ACL. |
| 1817 | |
| 1818 | show acl [<acl>] |
| 1819 | Dump info about acl converters. Without argument, the list of all available |
| 1820 | acls is returned. If a <acl> is specified, its contents are dumped. <acl> if |
| 1821 | the #<id> or <file>. The dump format is the same than the map even for the |
| 1822 | sample value. The data returned are not a list of available ACL, but are the |
| 1823 | list of all patterns composing any ACL. Many of these patterns can be shared |
| 1824 | with maps. |
| 1825 | |
| 1826 | show pools |
| 1827 | Dump the status of internal memory pools. This is useful to track memory |
| 1828 | usage when suspecting a memory leak for example. It does exactly the same |
| 1829 | as the SIGQUIT when running in foreground except that it does not flush |
| 1830 | the pools. |
| 1831 | |
| 1832 | show servers state [<backend>] |
| 1833 | Dump the state of the servers found in the running configuration. A backend |
| 1834 | name or identifier may be provided to limit the output to this backend only. |
| 1835 | |
| 1836 | The dump has the following format: |
| 1837 | - first line contains the format version (1 in this specification); |
| 1838 | - second line contains the column headers, prefixed by a sharp ('#'); |
| 1839 | - third line and next ones contain data; |
| 1840 | - each line starting by a sharp ('#') is considered as a comment. |
| 1841 | |
| 1842 | Since multiple versions of the ouptput may co-exist, below is the list of |
| 1843 | fields and their order per file format version : |
| 1844 | 1: |
| 1845 | be_id: Backend unique id. |
| 1846 | be_name: Backend label. |
| 1847 | srv_id: Server unique id (in the backend). |
| 1848 | srv_name: Server label. |
| 1849 | srv_addr: Server IP address. |
| 1850 | srv_op_state: Server operational state (UP/DOWN/...). |
| 1851 | In source code: SRV_ST_*. |
| 1852 | srv_admin_state: Server administrative state (MAINT/DRAIN/...). |
| 1853 | In source code: SRV_ADMF_*. |
| 1854 | srv_uweight: User visible server's weight. |
| 1855 | srv_iweight: Server's initial weight. |
| 1856 | srv_time_since_last_change: Time since last operational change. |
| 1857 | srv_check_status: Last health check status. |
| 1858 | srv_check_result: Last check result (FAILED/PASSED/...). |
| 1859 | In source code: CHK_RES_*. |
| 1860 | srv_check_health: Checks rise / fall current counter. |
| 1861 | srv_check_state: State of the check (ENABLED/PAUSED/...). |
| 1862 | In source code: CHK_ST_*. |
| 1863 | srv_agent_state: State of the agent check (ENABLED/PAUSED/...). |
| 1864 | In source code: CHK_ST_*. |
| 1865 | bk_f_forced_id: Flag to know if the backend ID is forced by |
| 1866 | configuration. |
| 1867 | srv_f_forced_id: Flag to know if the server's ID is forced by |
| 1868 | configuration. |
| 1869 | |
| 1870 | show sess |
| 1871 | Dump all known sessions. Avoid doing this on slow connections as this can |
| 1872 | be huge. This command is restricted and can only be issued on sockets |
| 1873 | configured for levels "operator" or "admin". |
| 1874 | |
| 1875 | show sess <id> |
| 1876 | Display a lot of internal information about the specified session identifier. |
| 1877 | This identifier is the first field at the beginning of the lines in the dumps |
| 1878 | of "show sess" (it corresponds to the session pointer). Those information are |
| 1879 | useless to most users but may be used by haproxy developers to troubleshoot a |
| 1880 | complex bug. The output format is intentionally not documented so that it can |
| 1881 | freely evolve depending on demands. You may find a description of all fields |
| 1882 | returned in src/dumpstats.c |
| 1883 | |
| 1884 | The special id "all" dumps the states of all sessions, which must be avoided |
| 1885 | as much as possible as it is highly CPU intensive and can take a lot of time. |
| 1886 | |
Willy Tarreau | 5d8b979 | 2016-03-11 11:09:34 +0100 | [diff] [blame] | 1887 | show stat [<iid> <type> <sid>] [typed] |
| 1888 | Dump statistics using the CSV format, or using the extended typed output |
| 1889 | format described in the section above if "typed" is passed after the other |
| 1890 | arguments. By passing <id>, <type> and <sid>, it is possible to dump only |
| 1891 | selected items : |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 1892 | - <iid> is a proxy ID, -1 to dump everything |
| 1893 | - <type> selects the type of dumpable objects : 1 for frontends, 2 for |
| 1894 | backends, 4 for servers, -1 for everything. These values can be ORed, |
| 1895 | for example: |
| 1896 | 1 + 2 = 3 -> frontend + backend. |
| 1897 | 1 + 2 + 4 = 7 -> frontend + backend + server. |
| 1898 | - <sid> is a server ID, -1 to dump everything from the selected proxy. |
| 1899 | |
| 1900 | Example : |
| 1901 | $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1 |
| 1902 | >>> Name: HAProxy |
| 1903 | Version: 1.4-dev2-49 |
| 1904 | Release_date: 2009/09/23 |
| 1905 | Nbproc: 1 |
| 1906 | Process_num: 1 |
| 1907 | (...) |
| 1908 | |
| 1909 | # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...) |
| 1910 | stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...) |
| 1911 | stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...) |
| 1912 | (...) |
| 1913 | www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...) |
| 1914 | |
| 1915 | $ |
| 1916 | |
Willy Tarreau | 5d8b979 | 2016-03-11 11:09:34 +0100 | [diff] [blame] | 1917 | In this example, two commands have been issued at once. That way it's easy to |
| 1918 | find which process the stats apply to in multi-process mode. This is not |
| 1919 | needed in the typed output format as the process number is reported on each |
| 1920 | line. Notice the empty line after the information output which marks the end |
| 1921 | of the first block. A similar empty line appears at the end of the second |
| 1922 | block (stats) so that the reader knows the output has not been truncated. |
| 1923 | |
| 1924 | When "typed" is specified, the output format is more suitable to monitoring |
| 1925 | tools because it provides numeric positions and indicates the type of each |
| 1926 | output field. Each value stands on its own line with process number, element |
| 1927 | number, nature, origin and scope. This same format is available via the HTTP |
| 1928 | stats by passing ";typed" after the URI. It is very important to note that in |
| 1929 | typed output format, the dump for a single object is contigous so that there |
| 1930 | is no need for a consumer to store everything at once. |
| 1931 | |
| 1932 | When using the typed output format, each line is made of 4 columns delimited |
| 1933 | by colons (':'). The first column is a dot-delimited series of 5 elements. The |
| 1934 | first element is a letter indicating the type of the object being described. |
| 1935 | At the moment the following object types are known : 'F' for a frontend, 'B' |
| 1936 | for a backend, 'L' for a listener, and 'S' for a server. The second element |
| 1937 | The second element is a positive integer representing the unique identifier of |
| 1938 | the proxy the object belongs to. It is equivalent to the "iid" column of the |
| 1939 | CSV output and matches the value in front of the optional "id" directive found |
| 1940 | in the frontend or backend section. The third element is a positive integer |
| 1941 | containing the unique object identifier inside the proxy, and corresponds to |
| 1942 | the "sid" column of the CSV output. ID 0 is reported when dumping a frontend |
| 1943 | or a backend. For a listener or a server, this corresponds to their respective |
| 1944 | ID inside the proxy. The fourth element is the numeric position of the field |
| 1945 | in the list (starting at zero). This position shall not change over time, but |
| 1946 | holes are to be expected, depending on build options or if some fields are |
| 1947 | deleted in the future. The fifth element is the field name as it appears in |
| 1948 | the CSV output. The sixth element is a positive integer and is the relative |
| 1949 | process number starting at 1. |
| 1950 | |
| 1951 | The rest of the line starting after the first colon follows the "typed output |
| 1952 | format" described in the section above. In short, the second column (after the |
| 1953 | first ':') indicates the origin, nature and scope of the variable. The third |
| 1954 | column indicates the type of the field, among "s32", "s64", "u32", "u64" and |
| 1955 | "str". Then the fourth column is the value itself, which the consumer knows |
| 1956 | how to parse thanks to column 3 and how to process thanks to column 2. |
| 1957 | |
| 1958 | Thus the overall line format in typed mode is : |
| 1959 | |
| 1960 | <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value> |
| 1961 | |
| 1962 | Here's an example of typed output format : |
| 1963 | |
| 1964 | $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1 |
| 1965 | F.2.0.0.pxname.1:MGP:str:private-frontend |
| 1966 | F.2.0.1.svname.1:MGP:str:FRONTEND |
| 1967 | F.2.0.8.bin.1:MGP:u64:0 |
| 1968 | F.2.0.9.bout.1:MGP:u64:0 |
| 1969 | F.2.0.40.hrsp_2xx.1:MGP:u64:0 |
| 1970 | L.2.1.0.pxname.1:MGP:str:private-frontend |
| 1971 | L.2.1.1.svname.1:MGP:str:sock-1 |
| 1972 | L.2.1.17.status.1:MGP:str:OPEN |
| 1973 | L.2.1.73.addr.1:MGP:str:0.0.0.0:8001 |
| 1974 | S.3.13.60.rtime.1:MCP:u32:0 |
| 1975 | S.3.13.61.ttime.1:MCP:u32:0 |
| 1976 | S.3.13.62.agent_status.1:MGP:str:L4TOUT |
| 1977 | S.3.13.64.agent_duration.1:MGP:u64:2001 |
| 1978 | S.3.13.65.check_desc.1:MCP:str:Layer4 timeout |
| 1979 | S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout |
| 1980 | S.3.13.67.check_rise.1:MCP:u32:2 |
| 1981 | S.3.13.68.check_fall.1:MCP:u32:3 |
| 1982 | S.3.13.69.check_health.1:SGP:u32:0 |
| 1983 | S.3.13.70.agent_rise.1:MaP:u32:1 |
| 1984 | S.3.13.71.agent_fall.1:SGP:u32:1 |
| 1985 | S.3.13.72.agent_health.1:SGP:u32:1 |
| 1986 | S.3.13.73.addr.1:MCP:str:1.255.255.255:8888 |
| 1987 | S.3.13.75.mode.1:MAP:str:http |
| 1988 | B.3.0.0.pxname.1:MGP:str:private-backend |
| 1989 | B.3.0.1.svname.1:MGP:str:BACKEND |
| 1990 | B.3.0.2.qcur.1:MGP:u32:0 |
| 1991 | B.3.0.3.qmax.1:MGP:u32:0 |
| 1992 | B.3.0.4.scur.1:MGP:u32:0 |
| 1993 | B.3.0.5.smax.1:MGP:u32:0 |
| 1994 | B.3.0.6.slim.1:MGP:u32:1000 |
| 1995 | B.3.0.55.lastsess.1:MMP:s32:-1 |
| 1996 | (...) |
| 1997 | |
| 1998 | In the typed format, the presence of the process ID at the end of the line |
| 1999 | makes it very easy to visually aggregate outputs from multiple processes, as |
| 2000 | show in the example below where each line appears for each process : |
| 2001 | |
| 2002 | $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \ |
| 2003 | echo show stat typed | socat /var/run/haproxy.sock2 - ) | \ |
| 2004 | sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n |
| 2005 | B.3.0.0.pxname.1:MGP:str:private-backend |
| 2006 | B.3.0.0.pxname.2:MGP:str:private-backend |
| 2007 | B.3.0.1.svname.1:MGP:str:BACKEND |
| 2008 | B.3.0.1.svname.2:MGP:str:BACKEND |
| 2009 | B.3.0.2.qcur.1:MGP:u32:0 |
| 2010 | B.3.0.2.qcur.2:MGP:u32:0 |
| 2011 | B.3.0.3.qmax.1:MGP:u32:0 |
| 2012 | B.3.0.3.qmax.2:MGP:u32:0 |
| 2013 | B.3.0.4.scur.1:MGP:u32:0 |
| 2014 | B.3.0.4.scur.2:MGP:u32:0 |
| 2015 | B.3.0.5.smax.1:MGP:u32:0 |
| 2016 | B.3.0.5.smax.2:MGP:u32:0 |
| 2017 | B.3.0.6.slim.1:MGP:u32:1000 |
| 2018 | B.3.0.6.slim.2:MGP:u32:1000 |
| 2019 | (...) |
Willy Tarreau | 44aed90 | 2015-10-13 14:45:29 +0200 | [diff] [blame] | 2020 | |
| 2021 | show stat resolvers [<resolvers section id>] |
| 2022 | Dump statistics for the given resolvers section, or all resolvers sections |
| 2023 | if no section is supplied. |
| 2024 | |
| 2025 | For each name server, the following counters are reported: |
| 2026 | sent: number of DNS requests sent to this server |
| 2027 | valid: number of DNS valid responses received from this server |
| 2028 | update: number of DNS responses used to update the server's IP address |
| 2029 | cname: number of CNAME responses |
| 2030 | cname_error: CNAME errors encountered with this server |
| 2031 | any_err: number of empty response (IE: server does not support ANY type) |
| 2032 | nx: non existent domain response received from this server |
| 2033 | timeout: how many time this server did not answer in time |
| 2034 | refused: number of requests refused by this server |
| 2035 | other: any other DNS errors |
| 2036 | invalid: invalid DNS response (from a protocol point of view) |
| 2037 | too_big: too big response |
| 2038 | outdated: number of response arrived too late (after an other name server) |
| 2039 | |
| 2040 | show table |
| 2041 | Dump general information on all known stick-tables. Their name is returned |
| 2042 | (the name of the proxy which holds them), their type (currently zero, always |
| 2043 | IP), their size in maximum possible number of entries, and the number of |
| 2044 | entries currently in use. |
| 2045 | |
| 2046 | Example : |
| 2047 | $ echo "show table" | socat stdio /tmp/sock1 |
| 2048 | >>> # table: front_pub, type: ip, size:204800, used:171454 |
| 2049 | >>> # table: back_rdp, type: ip, size:204800, used:0 |
| 2050 | |
| 2051 | show table <name> [ data.<type> <operator> <value> ] | [ key <key> ] |
| 2052 | Dump contents of stick-table <name>. In this mode, a first line of generic |
| 2053 | information about the table is reported as with "show table", then all |
| 2054 | entries are dumped. Since this can be quite heavy, it is possible to specify |
| 2055 | a filter in order to specify what entries to display. |
| 2056 | |
| 2057 | When the "data." form is used the filter applies to the stored data (see |
| 2058 | "stick-table" in section 4.2). A stored data type must be specified |
| 2059 | in <type>, and this data type must be stored in the table otherwise an |
| 2060 | error is reported. The data is compared according to <operator> with the |
| 2061 | 64-bit integer <value>. Operators are the same as with the ACLs : |
| 2062 | |
| 2063 | - eq : match entries whose data is equal to this value |
| 2064 | - ne : match entries whose data is not equal to this value |
| 2065 | - le : match entries whose data is less than or equal to this value |
| 2066 | - ge : match entries whose data is greater than or equal to this value |
| 2067 | - lt : match entries whose data is less than this value |
| 2068 | - gt : match entries whose data is greater than this value |
| 2069 | |
| 2070 | |
| 2071 | When the key form is used the entry <key> is shown. The key must be of the |
| 2072 | same type as the table, which currently is limited to IPv4, IPv6, integer, |
| 2073 | and string. |
| 2074 | |
| 2075 | Example : |
| 2076 | $ echo "show table http_proxy" | socat stdio /tmp/sock1 |
| 2077 | >>> # table: http_proxy, type: ip, size:204800, used:2 |
| 2078 | >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \ |
| 2079 | bytes_out_rate(60000)=187 |
| 2080 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 2081 | bytes_out_rate(60000)=191 |
| 2082 | |
| 2083 | $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1 |
| 2084 | >>> # table: http_proxy, type: ip, size:204800, used:2 |
| 2085 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 2086 | bytes_out_rate(60000)=191 |
| 2087 | |
| 2088 | $ echo "show table http_proxy data.conn_rate gt 5" | \ |
| 2089 | socat stdio /tmp/sock1 |
| 2090 | >>> # table: http_proxy, type: ip, size:204800, used:2 |
| 2091 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 2092 | bytes_out_rate(60000)=191 |
| 2093 | |
| 2094 | $ echo "show table http_proxy key 127.0.0.2" | \ |
| 2095 | socat stdio /tmp/sock1 |
| 2096 | >>> # table: http_proxy, type: ip, size:204800, used:2 |
| 2097 | >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \ |
| 2098 | bytes_out_rate(60000)=191 |
| 2099 | |
| 2100 | When the data criterion applies to a dynamic value dependent on time such as |
| 2101 | a bytes rate, the value is dynamically computed during the evaluation of the |
| 2102 | entry in order to decide whether it has to be dumped or not. This means that |
| 2103 | such a filter could match for some time then not match anymore because as |
| 2104 | time goes, the average event rate drops. |
| 2105 | |
| 2106 | It is possible to use this to extract lists of IP addresses abusing the |
| 2107 | service, in order to monitor them or even blacklist them in a firewall. |
| 2108 | Example : |
| 2109 | $ echo "show table http_proxy data.gpc0 gt 0" \ |
| 2110 | | socat stdio /tmp/sock1 \ |
| 2111 | | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt |
| 2112 | ( or | awk '/key/{ print a[split($2,a,"=")]; }' ) |
| 2113 | |
| 2114 | show tls-keys |
| 2115 | Dump all loaded TLS ticket keys. The TLS ticket key reference ID and the |
| 2116 | file from which the keys have been loaded is shown. Both of those can be |
| 2117 | used to update the TLS keys using "set ssl tls-key". |
| 2118 | |
| 2119 | shutdown frontend <frontend> |
| 2120 | Completely delete the specified frontend. All the ports it was bound to will |
| 2121 | be released. It will not be possible to enable the frontend anymore after |
| 2122 | this operation. This is intended to be used in environments where stopping a |
| 2123 | proxy is not even imaginable but a misconfigured proxy must be fixed. That |
| 2124 | way it's possible to release the port and bind it into another process to |
| 2125 | restore operations. The frontend will not appear at all on the stats page |
| 2126 | once it is terminated. |
| 2127 | |
| 2128 | The frontend may be specified either by its name or by its numeric ID, |
| 2129 | prefixed with a sharp ('#'). |
| 2130 | |
| 2131 | This command is restricted and can only be issued on sockets configured for |
| 2132 | level "admin". |
| 2133 | |
| 2134 | shutdown session <id> |
| 2135 | Immediately terminate the session matching the specified session identifier. |
| 2136 | This identifier is the first field at the beginning of the lines in the dumps |
| 2137 | of "show sess" (it corresponds to the session pointer). This can be used to |
| 2138 | terminate a long-running session without waiting for a timeout or when an |
| 2139 | endless transfer is ongoing. Such terminated sessions are reported with a 'K' |
| 2140 | flag in the logs. |
| 2141 | |
| 2142 | shutdown sessions server <backend>/<server> |
| 2143 | Immediately terminate all the sessions attached to the specified server. This |
| 2144 | can be used to terminate long-running sessions after a server is put into |
| 2145 | maintenance mode, for instance. Such terminated sessions are reported with a |
| 2146 | 'K' flag in the logs. |
| 2147 | |
Willy Tarreau | 2212e6a | 2015-10-13 14:40:55 +0200 | [diff] [blame] | 2148 | |
| 2149 | 10. Tricks for easier configuration management |
| 2150 | ---------------------------------------------- |
| 2151 | |
| 2152 | It is very common that two HAProxy nodes constituting a cluster share exactly |
| 2153 | the same configuration modulo a few addresses. Instead of having to maintain a |
| 2154 | duplicate configuration for each node, which will inevitably diverge, it is |
| 2155 | possible to include environment variables in the configuration. Thus multiple |
| 2156 | configuration may share the exact same file with only a few different system |
| 2157 | wide environment variables. This started in version 1.5 where only addresses |
| 2158 | were allowed to include environment variables, and 1.6 goes further by |
| 2159 | supporting environment variables everywhere. The syntax is the same as in the |
| 2160 | UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening |
| 2161 | curly brace ('{'), then the variable name followed by the closing brace ('}'). |
| 2162 | Except for addresses, environment variables are only interpreted in arguments |
| 2163 | surrounded with double quotes (this was necessary not to break existing setups |
| 2164 | using regular expressions involving the dollar symbol). |
| 2165 | |
| 2166 | Environment variables also make it convenient to write configurations which are |
| 2167 | expected to work on various sites where only the address changes. It can also |
| 2168 | permit to remove passwords from some configs. Example below where the the file |
| 2169 | "site1.env" file is sourced by the init script upon startup : |
| 2170 | |
| 2171 | $ cat site1.env |
| 2172 | LISTEN=192.168.1.1 |
| 2173 | CACHE_PFX=192.168.11 |
| 2174 | SERVER_PFX=192.168.22 |
| 2175 | LOGGER=192.168.33.1 |
| 2176 | STATSLP=admin:pa$$w0rd |
| 2177 | ABUSERS=/etc/haproxy/abuse.lst |
| 2178 | TIMEOUT=10s |
| 2179 | |
| 2180 | $ cat haproxy.cfg |
| 2181 | global |
| 2182 | log "${LOGGER}:514" local0 |
| 2183 | |
| 2184 | defaults |
| 2185 | mode http |
| 2186 | timeout client "${TIMEOUT}" |
| 2187 | timeout server "${TIMEOUT}" |
| 2188 | timeout connect 5s |
| 2189 | |
| 2190 | frontend public |
| 2191 | bind "${LISTEN}:80" |
| 2192 | http-request reject if { src -f "${ABUSERS}" } |
| 2193 | stats uri /stats |
| 2194 | stats auth "${STATSLP}" |
| 2195 | use_backend cache if { path_end .jpg .css .ico } |
| 2196 | default_backend server |
| 2197 | |
| 2198 | backend cache |
| 2199 | server cache1 "${CACHE_PFX}.1:18080" check |
| 2200 | server cache2 "${CACHE_PFX}.2:18080" check |
| 2201 | |
| 2202 | backend server |
| 2203 | server cache1 "${SERVER_PFX}.1:8080" check |
| 2204 | server cache2 "${SERVER_PFX}.2:8080" check |
| 2205 | |
| 2206 | |
| 2207 | 11. Well-known traps to avoid |
| 2208 | ----------------------------- |
| 2209 | |
| 2210 | Once in a while, someone reports that after a system reboot, the haproxy |
| 2211 | service wasn't started, and that once they start it by hand it works. Most |
| 2212 | often, these people are running a clustered IP address mechanism such as |
| 2213 | keepalived, to assign the service IP address to the master node only, and while |
| 2214 | it used to work when they used to bind haproxy to address 0.0.0.0, it stopped |
| 2215 | working after they bound it to the virtual IP address. What happens here is |
| 2216 | that when the service starts, the virtual IP address is not yet owned by the |
| 2217 | local node, so when HAProxy wants to bind to it, the system rejects this |
| 2218 | because it is not a local IP address. The fix doesn't consist in delaying the |
| 2219 | haproxy service startup (since it wouldn't stand a restart), but instead to |
| 2220 | properly configure the system to allow binding to non-local addresses. This is |
| 2221 | easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This |
| 2222 | is also needed in order to transparently intercept the IP traffic that passes |
| 2223 | through HAProxy for a specific target address. |
| 2224 | |
| 2225 | Multi-process configurations involving source port ranges may apparently seem |
| 2226 | to work but they will cause some random failures under high loads because more |
| 2227 | than one process may try to use the same source port to connect to the same |
| 2228 | server, which is not possible. The system will report an error and a retry will |
| 2229 | happen, picking another port. A high value in the "retries" parameter may hide |
| 2230 | the effect to a certain extent but this also comes with increased CPU usage and |
| 2231 | processing time. Logs will also report a certain number of retries. For this |
| 2232 | reason, port ranges should be avoided in multi-process configurations. |
| 2233 | |
| 2234 | Since HAProxy uses SO_REUSEPORT and supports having multiple independant |
| 2235 | processes bound to the same IP:port, during troubleshooting it can happen that |
| 2236 | an old process was not stopped before a new one was started. This provides |
| 2237 | absurd test results which tend to indicate that any change to the configuration |
| 2238 | is ignored. The reason is that in fact even the new process is restarted with a |
| 2239 | new configuration, the old one also gets some incoming connections and |
| 2240 | processes them, returning unexpected results. When in doubt, just stop the new |
| 2241 | process and try again. If it still works, it very likely means that an old |
| 2242 | process remains alive and has to be stopped. Linux's "netstat -lntp" is of good |
| 2243 | help here. |
| 2244 | |
| 2245 | When adding entries to an ACL from the command line (eg: when blacklisting a |
| 2246 | source address), it is important to keep in mind that these entries are not |
| 2247 | synchronized to the file and that if someone reloads the configuration, these |
| 2248 | updates will be lost. While this is often the desired effect (for blacklisting) |
| 2249 | it may not necessarily match expectations when the change was made as a fix for |
| 2250 | a problem. See the "add acl" action of the CLI interface. |
| 2251 | |
| 2252 | |
| 2253 | 12. Debugging and performance issues |
| 2254 | ------------------------------------ |
| 2255 | |
| 2256 | When HAProxy is started with the "-d" option, it will stay in the foreground |
| 2257 | and will print one line per event, such as an incoming connection, the end of a |
| 2258 | connection, and for each request or response header line seen. This debug |
| 2259 | output is emitted before the contents are processed, so they don't consider the |
| 2260 | local modifications. The main use is to show the request and response without |
| 2261 | having to run a network sniffer. The output is less readable when multiple |
| 2262 | connections are handled in parallel, though the "debug2ansi" and "debug2html" |
| 2263 | scripts found in the examples/ directory definitely help here by coloring the |
| 2264 | output. |
| 2265 | |
| 2266 | If a request or response is rejected because HAProxy finds it is malformed, the |
| 2267 | best thing to do is to connect to the CLI and issue "show errors", which will |
| 2268 | report the last captured faulty request and response for each frontend and |
| 2269 | backend, with all the necessary information to indicate precisely the first |
| 2270 | character of the input stream that was rejected. This is sometimes needed to |
| 2271 | prove to customers or to developers that a bug is present in their code. In |
| 2272 | this case it is often possible to relax the checks (but still keep the |
| 2273 | captures) using "option accept-invalid-http-request" or its equivalent for |
| 2274 | responses coming from the server "option accept-invalid-http-response". Please |
| 2275 | see the configuration manual for more details. |
| 2276 | |
| 2277 | Example : |
| 2278 | |
| 2279 | > show errors |
| 2280 | Total events captured on [13/Oct/2015:13:43:47.169] : 1 |
| 2281 | |
| 2282 | [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request |
| 2283 | backend <NONE> (#-1), server <NONE> (#-1), event #0 |
| 2284 | src 127.0.0.1:51981, session #0, session flags 0x00000080 |
| 2285 | HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000 |
| 2286 | HTTP chunk len 0 bytes, HTTP body len 0 bytes |
| 2287 | buffer flags 0x00808002, out 0 bytes, total 31 bytes |
| 2288 | pending 31 bytes, wrapping at 8040, error at position 13: |
| 2289 | |
| 2290 | 00000 GET /invalid request HTTP/1.1\r\n |
| 2291 | |
| 2292 | |
| 2293 | The output of "show info" on the CLI provides a number of useful information |
| 2294 | regarding the maximum connection rate ever reached, maximum SSL key rate ever |
| 2295 | reached, and in general all information which can help to explain temporary |
| 2296 | issues regarding CPU or memory usage. Example : |
| 2297 | |
| 2298 | > show info |
| 2299 | Name: HAProxy |
| 2300 | Version: 1.6-dev7-e32d18-17 |
| 2301 | Release_date: 2015/10/12 |
| 2302 | Nbproc: 1 |
| 2303 | Process_num: 1 |
| 2304 | Pid: 7949 |
| 2305 | Uptime: 0d 0h02m39s |
| 2306 | Uptime_sec: 159 |
| 2307 | Memmax_MB: 0 |
| 2308 | Ulimit-n: 120032 |
| 2309 | Maxsock: 120032 |
| 2310 | Maxconn: 60000 |
| 2311 | Hard_maxconn: 60000 |
| 2312 | CurrConns: 0 |
| 2313 | CumConns: 3 |
| 2314 | CumReq: 3 |
| 2315 | MaxSslConns: 0 |
| 2316 | CurrSslConns: 0 |
| 2317 | CumSslConns: 0 |
| 2318 | Maxpipes: 0 |
| 2319 | PipesUsed: 0 |
| 2320 | PipesFree: 0 |
| 2321 | ConnRate: 0 |
| 2322 | ConnRateLimit: 0 |
| 2323 | MaxConnRate: 1 |
| 2324 | SessRate: 0 |
| 2325 | SessRateLimit: 0 |
| 2326 | MaxSessRate: 1 |
| 2327 | SslRate: 0 |
| 2328 | SslRateLimit: 0 |
| 2329 | MaxSslRate: 0 |
| 2330 | SslFrontendKeyRate: 0 |
| 2331 | SslFrontendMaxKeyRate: 0 |
| 2332 | SslFrontendSessionReuse_pct: 0 |
| 2333 | SslBackendKeyRate: 0 |
| 2334 | SslBackendMaxKeyRate: 0 |
| 2335 | SslCacheLookups: 0 |
| 2336 | SslCacheMisses: 0 |
| 2337 | CompressBpsIn: 0 |
| 2338 | CompressBpsOut: 0 |
| 2339 | CompressBpsRateLim: 0 |
| 2340 | ZlibMemUsage: 0 |
| 2341 | MaxZlibMemUsage: 0 |
| 2342 | Tasks: 5 |
| 2343 | Run_queue: 1 |
| 2344 | Idle_pct: 100 |
| 2345 | node: wtap |
| 2346 | description: |
| 2347 | |
| 2348 | When an issue seems to randomly appear on a new version of HAProxy (eg: every |
| 2349 | second request is aborted, occasional crash, etc), it is worth trying to enable |
| 2350 | memory poisonning so that each call to malloc() is immediately followed by the |
| 2351 | filling of the memory area with a configurable byte. By default this byte is |
| 2352 | 0x50 (ASCII for 'P'), but any other byte can be used, including zero (which |
| 2353 | will have the same effect as a calloc() and which may make issues disappear). |
| 2354 | Memory poisonning is enabled on the command line using the "-dM" option. It |
| 2355 | slightly hurts performance and is not recommended for use in production. If |
| 2356 | an issue happens all the time with it or never happens when poisoonning uses |
| 2357 | byte zero, it clearly means you've found a bug and you definitely need to |
| 2358 | report it. Otherwise if there's no clear change, the problem it is not related. |
| 2359 | |
| 2360 | When debugging some latency issues, it is important to use both strace and |
| 2361 | tcpdump on the local machine, and another tcpdump on the remote system. The |
| 2362 | reason for this is that there are delays everywhere in the processing chain and |
| 2363 | it is important to know which one is causing latency to know where to act. In |
| 2364 | practice, the local tcpdump will indicate when the input data come in. Strace |
| 2365 | will indicate when haproxy receives these data (using recv/recvfrom). Warning, |
| 2366 | openssl uses read()/write() syscalls instead of recv()/send(). Strace will also |
| 2367 | show when haproxy sends the data, and tcpdump will show when the system sends |
| 2368 | these data to the interface. Then the external tcpdump will show when the data |
| 2369 | sent are really received (since the local one only shows when the packets are |
| 2370 | queued). The benefit of sniffing on the local system is that strace and tcpdump |
| 2371 | will use the same reference clock. Strace should be used with "-tts200" to get |
| 2372 | complete timestamps and report large enough chunks of data to read them. |
| 2373 | Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence |
| 2374 | numbers and complete timestamps. |
| 2375 | |
| 2376 | In practice, received data are almost always immediately received by haproxy |
| 2377 | (unless the machine has a saturated CPU or these data are invalid and not |
| 2378 | delivered). If these data are received but not sent, it generally is because |
| 2379 | the output buffer is saturated (ie: recipient doesn't consume the data fast |
| 2380 | enough). This can be confirmed by seeing that the polling doesn't notify of |
| 2381 | the ability to write on the output file descriptor for some time (it's often |
| 2382 | easier to spot in the strace output when the data finally leave and then roll |
| 2383 | back to see when the write event was notified). It generally matches an ACK |
| 2384 | received from the recipient, and detected by tcpdump. Once the data are sent, |
| 2385 | they may spend some time in the system doing nothing. Here again, the TCP |
| 2386 | congestion window may be limited and not allow these data to leave, waiting for |
| 2387 | an ACK to open the window. If the traffic is idle and the data take 40 ms or |
| 2388 | 200 ms to leave, it's a different issue (which is not an issue), it's the fact |
| 2389 | that the Nagle algorithm prevents empty packets from leaving immediately, in |
| 2390 | hope that they will be merged with subsequent data. HAProxy automatically |
| 2391 | disables Nagle in pure TCP mode and in tunnels. However it definitely remains |
| 2392 | enabled when forwarding an HTTP body (and this contributes to the performance |
| 2393 | improvement there by reducing the number of packets). Some HTTP non-compliant |
| 2394 | applications may be sensitive to the latency when delivering incomplete HTTP |
| 2395 | response messages. In this case you will have to enable "option http-no-delay" |
| 2396 | to disable Nagle in order to work around their design, keeping in mind that any |
| 2397 | other proxy in the chain may similarly be impacted. If tcpdump reports that data |
| 2398 | leave immediately but the other end doesn't see them quickly, it can mean there |
| 2399 | is a congestionned WAN link, a congestionned LAN with flow control enabled and |
| 2400 | preventing the data from leaving, or more commonly that HAProxy is in fact |
| 2401 | running in a virtual machine and that for whatever reason the hypervisor has |
| 2402 | decided that the data didn't need to be sent immediately. In virtualized |
| 2403 | environments, latency issues are almost always caused by the virtualization |
| 2404 | layer, so in order to save time, it's worth first comparing tcpdump in the VM |
| 2405 | and on the external components. Any difference has to be credited to the |
| 2406 | hypervisor and its accompanying drivers. |
| 2407 | |
| 2408 | When some TCP SACK segments are seen in tcpdump traces (using -vv), it always |
| 2409 | means that the side sending them has got the proof of a lost packet. While not |
| 2410 | seeing them doesn't mean there are no losses, seeing them definitely means the |
| 2411 | network is lossy. Losses are normal on a network, but at a rate where SACKs are |
| 2412 | not noticeable at the naked eye. If they appear a lot in the traces, it is |
| 2413 | worth investigating exactly what happens and where the packets are lost. HTTP |
| 2414 | doesn't cope well with TCP losses, which introduce huge latencies. |
| 2415 | |
| 2416 | The "netstat -i" command will report statistics per interface. An interface |
| 2417 | where the Rx-Ovr counter grows indicates that the system doesn't have enough |
| 2418 | resources to receive all incoming packets and that they're lost before being |
| 2419 | processed by the network driver. Rx-Drp indicates that some received packets |
| 2420 | were lost in the network stack because the application doesn't process them |
| 2421 | fast enough. This can happen during some attacks as well. Tx-Drp means that |
| 2422 | the output queues were full and packets had to be dropped. When using TCP it |
| 2423 | should be very rare, but will possibly indicte a saturated outgoing link. |
| 2424 | |
| 2425 | |
| 2426 | 13. Security considerations |
| 2427 | --------------------------- |
| 2428 | |
| 2429 | HAProxy is designed to run with very limited privileges. The standard way to |
| 2430 | use it is to isolate it into a chroot jail and to drop its privileges to a |
| 2431 | non-root user without any permissions inside this jail so that if any future |
| 2432 | vulnerability were to be discovered, its compromise would not affect the rest |
| 2433 | of the system. |
| 2434 | |
| 2435 | In order to perfom a chroot, it first needs to be started as a root user. It is |
| 2436 | pointless to build hand-made chroots to start the process there, these ones are |
| 2437 | painful to build, are never properly maintained and always contain way more |
| 2438 | bugs than the main file-system. And in case of compromise, the intruder can use |
| 2439 | the purposely built file-system. Unfortunately many administrators confuse |
| 2440 | "start as root" and "run as root", resulting in the uid change to be done prior |
| 2441 | to starting haproxy, and reducing the effective security restrictions. |
| 2442 | |
| 2443 | HAProxy will need to be started as root in order to : |
| 2444 | - adjust the file descriptor limits |
| 2445 | - bind to privileged port numbers |
| 2446 | - bind to a specific network interface |
| 2447 | - transparently listen to a foreign address |
| 2448 | - isolate itself inside the chroot jail |
| 2449 | - drop to another non-privileged UID |
| 2450 | |
| 2451 | HAProxy may require to be run as root in order to : |
| 2452 | - bind to an interface for outgoing connections |
| 2453 | - bind to privileged source ports for outgoing connections |
| 2454 | - transparently bind to a foreing address for outgoing connections |
| 2455 | |
| 2456 | Most users will never need the "run as root" case. But the "start as root" |
| 2457 | covers most usages. |
| 2458 | |
| 2459 | A safe configuration will have : |
| 2460 | |
| 2461 | - a chroot statement pointing to an empty location without any access |
| 2462 | permissions. This can be prepared this way on the UNIX command line : |
| 2463 | |
| 2464 | # mkdir /var/empty && chmod 0 /var/empty || echo "Failed" |
| 2465 | |
| 2466 | and referenced like this in the HAProxy configuration's global section : |
| 2467 | |
| 2468 | chroot /var/empty |
| 2469 | |
| 2470 | - both a uid/user and gid/group statements in the global section : |
| 2471 | |
| 2472 | user haproxy |
| 2473 | group haproxy |
| 2474 | |
| 2475 | - a stats socket whose mode, uid and gid are set to match the user and/or |
| 2476 | group allowed to access the CLI so that nobody may access it : |
| 2477 | |
| 2478 | stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600 |
| 2479 | |