| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 1 | PROMEX: A Prometheus exporter for HAProxy |
| 2 | ------------------------------------------- |
| 3 | |
| 4 | Prometheus is a monitoring and alerting system. More and more people use it to |
| 5 | monitor their environment (this is written February 2019). It collects metrics |
| 6 | from monitored targets by scraping metrics HTTP endpoints on these targets. For |
| 7 | HAProxy, The Prometheus team offically supports an exporter written in Go |
| 8 | (https://github.com/prometheus/haproxy_exporter). But it requires an extra |
| 9 | software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus |
| 10 | exporter for HAProxy. It was developed as a service and is directly available in |
| 11 | HAProxy, like the stats applet. |
| 12 | |
| 13 | However, PROMEX is not built by default with HAProxy. It is provided as an extra |
| 14 | component for everyone want to use it. So you need to explicity build HAProxy |
| 15 | with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance: |
| 16 | |
| Willy Tarreau | d254aa8 | 2019-06-14 18:40:48 +0200 | [diff] [blame] | 17 | > make TARGET=linux-glibc EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 18 | |
| 19 | if HAProxy provides the PROMEX service, the following build option will be |
| 20 | reported by the command "haproxy -vv": |
| 21 | |
| 22 | Built with the Prometheus exporter as a service |
| 23 | |
| 24 | To be used, it must be enabled in the configuration with an "http-request" rule |
| 25 | and the corresponding HTTP proxy must enable the HTX support. For instance: |
| 26 | |
| 27 | frontend test |
| 28 | mode http |
| 29 | ... |
| 30 | option http-use-htx |
| 31 | http-request use-service prometheus-exporter if { path /metrics } |
| 32 | ... |
| 33 | |
| 34 | |
| 35 | This service has been developed as a third-party component because it could |
| 36 | become obsolete, depending on how much time Prometheus will remain heavily |
| 37 | used. This is said with no ulterior motive of course. Prometheus is a great |
| 38 | software and I hope all the well for it. But we involve in a environment moving |
| 39 | quickly and a solution may be obvious today could be deprecated the next |
| 40 | year. And because PROMEX is not integrated by default into the HAProxy codebase, |
| 41 | it will need some interest to be actively supported. All contribution of any |
| 42 | kind are welcome. |
| 43 | |
| 44 | You must also be careful if you use with huge configurations. Unlike the stats |
| 45 | applet, all metrics are not grouped by service (proxy, listener or server). With |
| 46 | PROMEX, all lines for a given metric are provided as one single group. So |
| 47 | instead of collecting all metrics for a proxy before moving to the next one, we |
| 48 | must loop on all proxies for each metric. Same for the servers. Thus, it will |
| 49 | spend much more ressources to produce the Prometheus metrics than the CSV export |
| 50 | through the stats page. To give a comparison order, quick benchmarks shown that |
| 51 | a PROMEX dump is 5x slower and 20x more verbose than a CSV export. |
| 52 | |
| Christopher Faulet | 32d634f | 2019-11-18 14:47:08 +0100 | [diff] [blame^] | 53 | |
| 54 | metrics filtering |
| 55 | ------------------- |
| 56 | |
| 57 | It is possible to dynamically select the metrics to export if you don't use all |
| 58 | of them passing parameters in the query-string. |
| 59 | |
| 60 | * Filtering on scopes |
| 61 | |
| 62 | The metrics may be filtered by scopes. Multiple parameters with "scope" as name |
| 63 | may be passed in the query-string to filter exported metrics, with one of those |
| 64 | values: global, frontend, backend, server or '*' (means all). A scope parameter |
| 65 | with no value means to filter out all scopes (nothing is returned). The scope |
| 66 | parameters are parsed in their appearance order in the query-string. So an empty |
| 67 | scope will reset all scopes already parsed. But it can be overridden by |
| 68 | following scope parameters in the query-string. By default everything is |
| 69 | exported. Here are examples: |
| 70 | |
| 71 | /metrics?scope=server # ==> server metrics will be exported |
| 72 | /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported |
| 73 | /metrics?scope=*&scope= # ==> no metrics will be exported |
| 74 | /metrics?scope=&scope=global # ==> global metrics will be exported |
| 75 | |
| 76 | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 77 | Exported metrics |
| 78 | ------------------ |
| 79 | |
| 80 | * Globals metrics |
| 81 | |
| 82 | +------------------------------------------------+-------------------------------------------------------------------------------+ |
| 83 | | Metric name | Description | |
| 84 | +------------------------------------------------+-------------------------------------------------------------------------------+ |
| 85 | | haproxy_process_nbthread | Configured number of threads. | |
| 86 | | haproxy_process_nbproc | Configured number of processes. | |
| 87 | | haproxy_process_relative_process_id | Relative process id, starting at 1. | |
| Christopher Faulet | 8c8e4b1 | 2019-04-18 10:15:15 +0200 | [diff] [blame] | 88 | | haproxy_process_start_time_seconds | Start time in seconds. | |
| 89 | | haproxy_process_max_memory_bytes | Per-process memory limit (in bytes); 0=unset. | |
| 90 | | haproxy_process_pool_allocated_bytes | Total amount of memory allocated in pools (in bytes). | |
| 91 | | haproxy_process_pool_used_bytes | Total amount of memory used in pools (in bytes). | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 92 | | haproxy_process_pool_failures_total | Total number of failed pool allocations. | |
| 93 | | haproxy_process_max_fds | Maximum number of open file descriptors; 0=unset. | |
| 94 | | haproxy_process_max_sockets | Maximum numer of open sockets. | |
| 95 | | haproxy_process_max_connections | Maximum number of concurrent connections. | |
| 96 | | haproxy_process_hard_max_connections | Initial Maximum number of concurrent connections. | |
| 97 | | haproxy_process_current_connections | Number of active sessions. | |
| Christopher Faulet | 8c8e4b1 | 2019-04-18 10:15:15 +0200 | [diff] [blame] | 98 | | haproxy_process_connections_total | Total number of created sessions. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 99 | | haproxy_process_requests_total | Total number of requests (TCP or HTTP). | |
| 100 | | haproxy_process_max_ssl_connections | Configured maximum number of concurrent SSL connections. | |
| 101 | | haproxy_process_current_ssl_connections | Number of opened SSL connections. | |
| 102 | | haproxy_process_ssl_connections_total | Total number of opened SSL connections. | |
| 103 | | haproxy_process_max_pipes | Configured maximum number of pipes. | |
| 104 | | haproxy_process_pipes_used_total | Number of pipes in used. | |
| 105 | | haproxy_process_pipes_free_total | Number of pipes unused. | |
| 106 | | haproxy_process_current_connection_rate | Current number of connections per second over last elapsed second. | |
| 107 | | haproxy_process_limit_connection_rate | Configured maximum number of connections per second. | |
| 108 | | haproxy_process_max_connection_rate | Maximum observed number of connections per second. | |
| 109 | | haproxy_process_current_session_rate | Current number of sessions per second over last elapsed second. | |
| 110 | | haproxy_process_limit_session_rate | Configured maximum number of sessions per second. | |
| 111 | | haproxy_process_max_session_rate | Maximum observed number of sessions per second. | |
| 112 | | haproxy_process_current_ssl_rate | Current number of SSL sessions per second over last elapsed second. | |
| 113 | | haproxy_process_limit_ssl_rate | Configured maximum number of SSL sessions per second. | |
| 114 | | haproxy_process_max_ssl_rate | Maximum observed number of SSL sessions per second. | |
| 115 | | haproxy_process_current_frontend_ssl_key_rate | Current frontend SSL Key computation per second over last elapsed second. | |
| 116 | | haproxy_process_max_frontend_ssl_key_rate | Maximum observed frontend SSL Key computation per second. | |
| 117 | | haproxy_process_frontent_ssl_reuse | SSL session reuse ratio (percent). | |
| 118 | | haproxy_process_current_backend_ssl_key_rate | Current backend SSL Key computation per second over last elapsed second. | |
| 119 | | haproxy_process_max_backend_ssl_key_rate | Maximum observed backend SSL Key computation per second. | |
| Christopher Faulet | 8c8e4b1 | 2019-04-18 10:15:15 +0200 | [diff] [blame] | 120 | | haproxy_process_ssl_cache_lookups_total | Total number of SSL session cache lookups. | |
| 121 | | haproxy_process_ssl_cache_misses_total | Total number of SSL session cache misses. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 122 | | haproxy_process_http_comp_bytes_in_total | Number of bytes per second over last elapsed second, before http compression. | |
| 123 | | haproxy_process_http_comp_bytes_out_total | Number of bytes per second over last elapsed second, after http compression. | |
| 124 | | haproxy_process_limit_http_comp | Configured maximum input compression rate in bytes. | |
| 125 | | haproxy_process_current_zlib_memory | Current memory used for zlib in bytes. | |
| 126 | | haproxy_process_max_zlib_memory | Configured maximum amount of memory for zlib in bytes. | |
| 127 | | haproxy_process_current_tasks | Current number of tasks. | |
| 128 | | haproxy_process_current_run_queue | Current number of tasks in the run-queue. | |
| 129 | | haproxy_process_idle_time_percent | Idle to total ratio over last sample (percent). | |
| 130 | | haproxy_process_stopping | Non zero means stopping in progress. | |
| 131 | | haproxy_process_jobs | Current number of active jobs (listeners, sessions, open devices). | |
| 132 | | haproxy_process_unstoppable_jobs | Current number of active jobs that can't be stopped during a soft stop. | |
| 133 | | haproxy_process_listeners | Current number of active listeners. | |
| 134 | | haproxy_process_active_peers | Current number of active peers. | |
| 135 | | haproxy_process_connected_peers | Current number of connected peers. | |
| 136 | | haproxy_process_dropped_logs_total | Total number of dropped logs. | |
| 137 | | haproxy_process_busy_polling_enabled | Non zero if the busy polling is enabled. | |
| 138 | +------------------------------------------------+-------------------------------------------------------------------------------+ |
| 139 | |
| 140 | * Frontend metrics |
| 141 | |
| 142 | +-------------------------------------------------+------------------------------------------------------------------------------+ |
| 143 | | Metric name | Description | |
| 144 | +-------------------------------------------------+------------------------------------------------------------------------------+ |
| 145 | | haproxy_frontend_status | Current status of the service. | |
| 146 | | haproxy_frontend_current_sessions | Current number of active sessions. | |
| 147 | | haproxy_frontend_max_sessions | Maximum observed number of active sessions. | |
| 148 | | haproxy_frontend_limit_sessions | Configured session limit. | |
| 149 | | haproxy_frontend_sessions_total | Total number of sessions. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 150 | | haproxy_frontend_limit_session_rate | Configured limit on new sessions per second. | |
| 151 | | haproxy_frontend_max_session_rate | Maximum observed number of sessions per second. | |
| 152 | | haproxy_frontend_connections_rate_current | Current number of connections per second over the last elapsed second. | |
| 153 | | haproxy_frontend_connections_rate_max | Maximum observed number of connections per second. | |
| 154 | | haproxy_frontend_connections_total | Total number of connections. | |
| 155 | | haproxy_frontend_bytes_in_total | Current total of incoming bytes. | |
| 156 | | haproxy_frontend_bytes_out_total | Current total of outgoing bytes. | |
| 157 | | haproxy_frontend_requests_denied_total | Total number of denied requests. | |
| 158 | | haproxy_frontend_responses_denied_total | Total number of denied responses. | |
| 159 | | haproxy_frontend_request_errors_total | Total number of request errors. | |
| 160 | | haproxy_frontend_denied_connections_total | Total number of requests denied by "tcp-request connection" rules. | |
| 161 | | haproxy_frontend_denied_sessions_total | Total number of requests denied by "tcp-request session" rules. | |
| 162 | | haproxy_frontend_failed_header_rewriting_total | Total number of failed header rewriting warnings. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 163 | | haproxy_frontend_http_requests_rate_max | Maximum observed number of HTTP requests per second. | |
| 164 | | haproxy_frontend_http_requests_total | Total number of HTTP requests received. | |
| 165 | | haproxy_frontend_http_responses_total | Total number of HTTP responses. | |
| 166 | | haproxy_frontend_intercepted_requests_total | Total number of intercepted HTTP requests. | |
| 167 | | haproxy_frontend_http_cache_lookups_total | Total number of HTTP cache lookups. | |
| 168 | | haproxy_frontend_http_cache_hits_total | Total number of HTTP cache hits. | |
| 169 | | haproxy_frontend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. | |
| 170 | | haproxy_frontend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. | |
| 171 | | haproxy_frontend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). | |
| 172 | | haproxy_frontend_http_comp_responses_total | Total number of HTTP responses that were compressed. | |
| 173 | +-------------------------------------------------+------------------------------------------------------------------------------+ |
| 174 | |
| Rick Rackow | a045ade | 2019-10-08 11:28:06 +0200 | [diff] [blame] | 175 | * Backend metrics |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 176 | |
| 177 | +-----------------------------------------------------+--------------------------------------------------------------------------+ |
| 178 | | Metric name | Description | |
| 179 | +-----------------------------------------------------+--------------------------------------------------------------------------+ |
| 180 | | haproxy_backend_status | Current status of the service. | |
| 181 | | haproxy_backend_current_sessions | Current number of active sessions. | |
| 182 | | haproxy_backend_max_sessions | Maximum observed number of active sessions. | |
| 183 | | haproxy_backend_limit_sessions | Configured session limit. | |
| 184 | | haproxy_backend_sessions_total | Total number of sessions. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 185 | | haproxy_backend_max_session_rate | Maximum observed number of sessions per second. | |
| 186 | | haproxy_backend_last_session_seconds | Number of seconds since last session assigned to server/backend. | |
| 187 | | haproxy_backend_current_queue | Current number of queued requests. | |
| 188 | | haproxy_backend_max_queue | Maximum observed number of queued requests. | |
| 189 | | haproxy_backend_connection_attempts_total | Total number of connection establishment attempts. | |
| 190 | | haproxy_backend_connection_reuses_total | Total number of connection reuses. | |
| 191 | | haproxy_backend_bytes_in_total | Current total of incoming bytes. | |
| 192 | | haproxy_backend_bytes_out_total | Current total of outgoing bytes. | |
| Christopher Faulet | 3388fd2 | 2019-11-08 15:12:29 +0100 | [diff] [blame] | 193 | | haproxy_backend_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. | |
| 194 | | haproxy_backend_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. | |
| 195 | | haproxy_backend_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) | |
| 196 | | haproxy_backend_total_time_average_seconds | Avg. total time for last 1024 successful connections. | |
| Christopher Faulet | 5df597a | 2019-11-08 15:05:31 +0100 | [diff] [blame] | 197 | | haproxy_backend_max_queue_time_seconds | Maximum observed queue time. | |
| 198 | | haproxy_backend_max_connect_time_seconds | Maximum observed connect time. | |
| Christopher Faulet | 3388fd2 | 2019-11-08 15:12:29 +0100 | [diff] [blame] | 199 | | haproxy_backend_max_response_time_seconds | Maximum observed response time. (0 for TCP) | |
| Christopher Faulet | 5df597a | 2019-11-08 15:05:31 +0100 | [diff] [blame] | 200 | | haproxy_backend_max_total_time_seconds | Maximum observed total time. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 201 | | haproxy_backend_requests_denied_total | Total number of denied requests. | |
| 202 | | haproxy_backend_responses_denied_total | Total number of denied responses. | |
| 203 | | haproxy_backend_connection_errors_total | Total number of connection errors. | |
| 204 | | haproxy_backend_response_errors_total | Total number of response errors. | |
| 205 | | haproxy_backend_retry_warnings_total | Total number of retry warnings. | |
| 206 | | haproxy_backend_redispatch_warnings_total | Total number of redispatch warnings. | |
| 207 | | haproxy_backend_failed_header_rewriting_total | Total number of failed header rewriting warnings. | |
| 208 | | haproxy_backend_client_aborts_total | Total number of data transfers aborted by the client. | |
| 209 | | haproxy_backend_server_aborts_total | Total number of data transfers aborted by the server. | |
| 210 | | haproxy_backend_weight | Service weight. | |
| 211 | | haproxy_backend_active_servers | Current number of active servers. | |
| 212 | | haproxy_backend_backup_servers | Current number of backup servers. | |
| 213 | | haproxy_backend_check_up_down_total | Total number of UP->DOWN transitions. | |
| 214 | | haproxy_backend_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. | |
| 215 | | haproxy_backend_downtime_seconds_total | Total downtime (in seconds) for the service. | |
| 216 | | haproxy_backend_loadbalanced_total | Total number of times a service was selected. | |
| 217 | | haproxy_backend_http_requests_total | Total number of HTTP requests received. | |
| 218 | | haproxy_backend_http_responses_total | Total number of HTTP responses. | |
| 219 | | haproxy_backend_http_cache_lookups_total | Total number of HTTP cache lookups. | |
| 220 | | haproxy_backend_http_cache_hits_total | Total number of HTTP cache hits. | |
| 221 | | haproxy_backend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. | |
| 222 | | haproxy_backend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. | |
| 223 | | haproxy_backend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). | |
| 224 | | haproxy_backend_http_comp_responses_total | Total number of HTTP responses that were compressed. | |
| 225 | +-----------------------------------------------------+--------------------------------------------------------------------------+ |
| 226 | |
| 227 | * Server metrics |
| 228 | |
| 229 | +----------------------------------------------------+---------------------------------------------------------------------------+ |
| 230 | | Metric name | Description | |
| 231 | +----------------------------------------------------+---------------------------------------------------------------------------+ |
| 232 | | haproxy_server_status | Current status of the service. | |
| 233 | | haproxy_server_current_sessions | Current number of active sessions. | |
| 234 | | haproxy_server_max_sessions | Maximum observed number of active sessions. | |
| 235 | | haproxy_server_limit_sessions | Configured session limit. | |
| 236 | | haproxy_server_sessions_total | Total number of sessions. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 237 | | haproxy_server_max_session_rate | Maximum observed number of sessions per second. | |
| 238 | | haproxy_server_last_session_seconds | Number of seconds since last session assigned to server/backend. | |
| 239 | | haproxy_server_current_queue | Current number of queued requests. | |
| 240 | | haproxy_server_max_queue | Maximum observed number of queued requests. | |
| 241 | | haproxy_server_queue_limit | Configured maxqueue for the server (0 meaning no limit). | |
| 242 | | haproxy_server_bytes_in_total | Current total of incoming bytes. | |
| 243 | | haproxy_server_bytes_out_total | Current total of outgoing bytes. | |
| Christopher Faulet | 3388fd2 | 2019-11-08 15:12:29 +0100 | [diff] [blame] | 244 | | haproxy_server_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. | |
| 245 | | haproxy_server_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. | |
| 246 | | haproxy_server_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) | |
| 247 | | haproxy_server_total_time_average_seconds | Avg. total time for last 1024 successful connections. | |
| Christopher Faulet | 5df597a | 2019-11-08 15:05:31 +0100 | [diff] [blame] | 248 | | haproxy_server_max_queue_time_seconds | Maximum observed queue time. | |
| 249 | | haproxy_server_max_connect_time_seconds | Maximum observed connect time. | |
| Christopher Faulet | 3388fd2 | 2019-11-08 15:12:29 +0100 | [diff] [blame] | 250 | | haproxy_server_max_response_time_seconds | Maximum observed response time. (0 for TCP) | |
| Christopher Faulet | 5df597a | 2019-11-08 15:05:31 +0100 | [diff] [blame] | 251 | | haproxy_server_max_total_time_seconds | Maximum observed total time. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 252 | | haproxy_server_connection_attempts_total | Total number of connection establishment attempts. | |
| 253 | | haproxy_server_connection_reuses_total | Total number of connection reuses. | |
| 254 | | haproxy_server_responses_denied_total | Total number of denied responses. | |
| 255 | | haproxy_server_connection_errors_total | Total number of connection errors. | |
| 256 | | haproxy_server_response_errors_total | Total number of response errors. | |
| 257 | | haproxy_server_retry_warnings_total | Total number of retry warnings. | |
| 258 | | haproxy_server_redispatch_warnings_total | Total number of redispatch warnings. | |
| 259 | | haproxy_server_failed_header_rewriting_total | Total number of failed header rewriting warnings. | |
| 260 | | haproxy_server_client_aborts_total | Total number of data transfers aborted by the client. | |
| 261 | | haproxy_server_server_aborts_total | Total number of data transfers aborted by the server. | |
| 262 | | haproxy_server_weight | Service weight. | |
| 263 | | haproxy_server_check_failures_total | Total number of failed check (Only when the server is up). | |
| 264 | | haproxy_server_check_up_down_total | Total number of UP->DOWN transitions. | |
| 265 | | haproxy_server_downtime_seconds_total | Total downtime (in seconds) for the service. | |
| 266 | | haproxy_server_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. | |
| 267 | | haproxy_server_current_throttle | Current throttle percentage for the server, when slowstart is active. | |
| 268 | | haproxy_server_loadbalanced_total | Total number of times a service was selected. | |
| 269 | | haproxy_server_http_responses_total | Total number of HTTP responses. | |
| Christopher Faulet | bd767f7 | 2019-11-08 15:24:32 +0100 | [diff] [blame] | 270 | | haproxy_server_idle_connections_current | Current number of idle connections available for reuse. | |
| 271 | | haproxy_server_idle_connections_limit | Limit on the number of available idle connections. | |
| Christopher Faulet | f959d08 | 2019-02-07 15:38:42 +0100 | [diff] [blame] | 272 | +----------------------------------------------------+---------------------------------------------------------------------------+ |