blob: 1c5a9924108a8cb6eb9b1573a6d577bf966d863b [file] [log] [blame]
Christopher Fauletf959d082019-02-07 15:38:42 +01001PROMEX: A Prometheus exporter for HAProxy
2-------------------------------------------
3
4Prometheus is a monitoring and alerting system. More and more people use it to
5monitor their environment (this is written February 2019). It collects metrics
6from monitored targets by scraping metrics HTTP endpoints on these targets. For
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +05007HAProxy, The Prometheus team officially supports an exporter written in Go
Christopher Fauletf959d082019-02-07 15:38:42 +01008(https://github.com/prometheus/haproxy_exporter). But it requires an extra
9software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
10exporter for HAProxy. It was developed as a service and is directly available in
11HAProxy, like the stats applet.
12
13However, PROMEX is not built by default with HAProxy. It is provided as an extra
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +050014component for everyone want to use it. So you need to explicitly build HAProxy
Christopher Fauletf959d082019-02-07 15:38:42 +010015with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance:
16
Willy Tarreaud254aa82019-06-14 18:40:48 +020017 > make TARGET=linux-glibc EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
Christopher Fauletf959d082019-02-07 15:38:42 +010018
19if HAProxy provides the PROMEX service, the following build option will be
20reported by the command "haproxy -vv":
21
22 Built with the Prometheus exporter as a service
23
24To be used, it must be enabled in the configuration with an "http-request" rule
25and the corresponding HTTP proxy must enable the HTX support. For instance:
26
27 frontend test
28 mode http
29 ...
30 option http-use-htx
31 http-request use-service prometheus-exporter if { path /metrics }
32 ...
33
34
35This service has been developed as a third-party component because it could
36become obsolete, depending on how much time Prometheus will remain heavily
37used. This is said with no ulterior motive of course. Prometheus is a great
38software and I hope all the well for it. But we involve in a environment moving
39quickly and a solution may be obvious today could be deprecated the next
40year. And because PROMEX is not integrated by default into the HAProxy codebase,
41it will need some interest to be actively supported. All contribution of any
42kind are welcome.
43
44You must also be careful if you use with huge configurations. Unlike the stats
45applet, all metrics are not grouped by service (proxy, listener or server). With
46PROMEX, all lines for a given metric are provided as one single group. So
47instead of collecting all metrics for a proxy before moving to the next one, we
48must loop on all proxies for each metric. Same for the servers. Thus, it will
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +050049spend much more resources to produce the Prometheus metrics than the CSV export
Christopher Fauletf959d082019-02-07 15:38:42 +010050through the stats page. To give a comparison order, quick benchmarks shown that
51a PROMEX dump is 5x slower and 20x more verbose than a CSV export.
52
Christopher Faulet78407ce2019-11-18 14:47:08 +010053
54metrics filtering
55-------------------
56
57It is possible to dynamically select the metrics to export if you don't use all
58of them passing parameters in the query-string.
59
60* Filtering on scopes
61
62The metrics may be filtered by scopes. Multiple parameters with "scope" as name
63may be passed in the query-string to filter exported metrics, with one of those
64values: global, frontend, backend, server or '*' (means all). A scope parameter
65with no value means to filter out all scopes (nothing is returned). The scope
66parameters are parsed in their appearance order in the query-string. So an empty
67scope will reset all scopes already parsed. But it can be overridden by
68following scope parameters in the query-string. By default everything is
69exported. Here are examples:
70
71 /metrics?scope=server # ==> server metrics will be exported
72 /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
73 /metrics?scope=*&scope= # ==> no metrics will be exported
74 /metrics?scope=&scope=global # ==> global metrics will be exported
75
Christopher Fauleteba22942019-11-19 14:18:24 +010076* Filtering on servers state
77
78It is possible to exclude from returned metrics all servers in maintenance mode
79passing the parameter "no-maint" in the query-string. This parameter may help to
80solve performance issues of configuration that use the server templates to
81manage dynamic provisionning. Note there is no consistency check on the servers
82state. So, if the state of a server changes while the exporter is running, only
83a part of the metrics for this server will be dumped.
Christopher Faulet78407ce2019-11-18 14:47:08 +010084
Christopher Fauletf959d082019-02-07 15:38:42 +010085Exported metrics
86------------------
87
88* Globals metrics
89
90+------------------------------------------------+-------------------------------------------------------------------------------+
91| Metric name | Description |
92+------------------------------------------------+-------------------------------------------------------------------------------+
93| haproxy_process_nbthread | Configured number of threads. |
94| haproxy_process_nbproc | Configured number of processes. |
95| haproxy_process_relative_process_id | Relative process id, starting at 1. |
Christopher Faulet8c8e4b12019-04-18 10:15:15 +020096| haproxy_process_start_time_seconds | Start time in seconds. |
97| haproxy_process_max_memory_bytes | Per-process memory limit (in bytes); 0=unset. |
98| haproxy_process_pool_allocated_bytes | Total amount of memory allocated in pools (in bytes). |
99| haproxy_process_pool_used_bytes | Total amount of memory used in pools (in bytes). |
Christopher Fauletf959d082019-02-07 15:38:42 +0100100| haproxy_process_pool_failures_total | Total number of failed pool allocations. |
101| haproxy_process_max_fds | Maximum number of open file descriptors; 0=unset. |
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +0500102| haproxy_process_max_sockets | Maximum number of open sockets. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100103| haproxy_process_max_connections | Maximum number of concurrent connections. |
104| haproxy_process_hard_max_connections | Initial Maximum number of concurrent connections. |
105| haproxy_process_current_connections | Number of active sessions. |
Christopher Faulet8c8e4b12019-04-18 10:15:15 +0200106| haproxy_process_connections_total | Total number of created sessions. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100107| haproxy_process_requests_total | Total number of requests (TCP or HTTP). |
108| haproxy_process_max_ssl_connections | Configured maximum number of concurrent SSL connections. |
109| haproxy_process_current_ssl_connections | Number of opened SSL connections. |
110| haproxy_process_ssl_connections_total | Total number of opened SSL connections. |
111| haproxy_process_max_pipes | Configured maximum number of pipes. |
112| haproxy_process_pipes_used_total | Number of pipes in used. |
113| haproxy_process_pipes_free_total | Number of pipes unused. |
114| haproxy_process_current_connection_rate | Current number of connections per second over last elapsed second. |
115| haproxy_process_limit_connection_rate | Configured maximum number of connections per second. |
116| haproxy_process_max_connection_rate | Maximum observed number of connections per second. |
117| haproxy_process_current_session_rate | Current number of sessions per second over last elapsed second. |
118| haproxy_process_limit_session_rate | Configured maximum number of sessions per second. |
119| haproxy_process_max_session_rate | Maximum observed number of sessions per second. |
120| haproxy_process_current_ssl_rate | Current number of SSL sessions per second over last elapsed second. |
121| haproxy_process_limit_ssl_rate | Configured maximum number of SSL sessions per second. |
122| haproxy_process_max_ssl_rate | Maximum observed number of SSL sessions per second. |
123| haproxy_process_current_frontend_ssl_key_rate | Current frontend SSL Key computation per second over last elapsed second. |
124| haproxy_process_max_frontend_ssl_key_rate | Maximum observed frontend SSL Key computation per second. |
125| haproxy_process_frontent_ssl_reuse | SSL session reuse ratio (percent). |
126| haproxy_process_current_backend_ssl_key_rate | Current backend SSL Key computation per second over last elapsed second. |
127| haproxy_process_max_backend_ssl_key_rate | Maximum observed backend SSL Key computation per second. |
Christopher Faulet8c8e4b12019-04-18 10:15:15 +0200128| haproxy_process_ssl_cache_lookups_total | Total number of SSL session cache lookups. |
129| haproxy_process_ssl_cache_misses_total | Total number of SSL session cache misses. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100130| haproxy_process_http_comp_bytes_in_total | Number of bytes per second over last elapsed second, before http compression. |
131| haproxy_process_http_comp_bytes_out_total | Number of bytes per second over last elapsed second, after http compression. |
132| haproxy_process_limit_http_comp | Configured maximum input compression rate in bytes. |
133| haproxy_process_current_zlib_memory | Current memory used for zlib in bytes. |
134| haproxy_process_max_zlib_memory | Configured maximum amount of memory for zlib in bytes. |
135| haproxy_process_current_tasks | Current number of tasks. |
136| haproxy_process_current_run_queue | Current number of tasks in the run-queue. |
137| haproxy_process_idle_time_percent | Idle to total ratio over last sample (percent). |
138| haproxy_process_stopping | Non zero means stopping in progress. |
139| haproxy_process_jobs | Current number of active jobs (listeners, sessions, open devices). |
140| haproxy_process_unstoppable_jobs | Current number of active jobs that can't be stopped during a soft stop. |
141| haproxy_process_listeners | Current number of active listeners. |
142| haproxy_process_active_peers | Current number of active peers. |
143| haproxy_process_connected_peers | Current number of connected peers. |
144| haproxy_process_dropped_logs_total | Total number of dropped logs. |
145| haproxy_process_busy_polling_enabled | Non zero if the busy polling is enabled. |
146+------------------------------------------------+-------------------------------------------------------------------------------+
147
148* Frontend metrics
149
150+-------------------------------------------------+------------------------------------------------------------------------------+
151| Metric name | Description |
152+-------------------------------------------------+------------------------------------------------------------------------------+
153| haproxy_frontend_status | Current status of the service. |
154| haproxy_frontend_current_sessions | Current number of active sessions. |
155| haproxy_frontend_max_sessions | Maximum observed number of active sessions. |
156| haproxy_frontend_limit_sessions | Configured session limit. |
157| haproxy_frontend_sessions_total | Total number of sessions. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100158| haproxy_frontend_limit_session_rate | Configured limit on new sessions per second. |
159| haproxy_frontend_max_session_rate | Maximum observed number of sessions per second. |
160| haproxy_frontend_connections_rate_current | Current number of connections per second over the last elapsed second. |
161| haproxy_frontend_connections_rate_max | Maximum observed number of connections per second. |
162| haproxy_frontend_connections_total | Total number of connections. |
163| haproxy_frontend_bytes_in_total | Current total of incoming bytes. |
164| haproxy_frontend_bytes_out_total | Current total of outgoing bytes. |
165| haproxy_frontend_requests_denied_total | Total number of denied requests. |
166| haproxy_frontend_responses_denied_total | Total number of denied responses. |
167| haproxy_frontend_request_errors_total | Total number of request errors. |
168| haproxy_frontend_denied_connections_total | Total number of requests denied by "tcp-request connection" rules. |
169| haproxy_frontend_denied_sessions_total | Total number of requests denied by "tcp-request session" rules. |
170| haproxy_frontend_failed_header_rewriting_total | Total number of failed header rewriting warnings. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100171| haproxy_frontend_http_requests_rate_max | Maximum observed number of HTTP requests per second. |
172| haproxy_frontend_http_requests_total | Total number of HTTP requests received. |
173| haproxy_frontend_http_responses_total | Total number of HTTP responses. |
174| haproxy_frontend_intercepted_requests_total | Total number of intercepted HTTP requests. |
175| haproxy_frontend_http_cache_lookups_total | Total number of HTTP cache lookups. |
176| haproxy_frontend_http_cache_hits_total | Total number of HTTP cache hits. |
177| haproxy_frontend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. |
178| haproxy_frontend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. |
179| haproxy_frontend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). |
180| haproxy_frontend_http_comp_responses_total | Total number of HTTP responses that were compressed. |
Christopher Faulete4a2c8d2019-12-16 14:44:01 +0100181| haproxy_frontend_internal_errors_total | Total number of internal errors. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100182+-------------------------------------------------+------------------------------------------------------------------------------+
183
Rick Rackow35efbe22019-10-08 11:28:06 +0200184* Backend metrics
Christopher Fauletf959d082019-02-07 15:38:42 +0100185
186+-----------------------------------------------------+--------------------------------------------------------------------------+
187| Metric name | Description |
188+-----------------------------------------------------+--------------------------------------------------------------------------+
189| haproxy_backend_status | Current status of the service. |
190| haproxy_backend_current_sessions | Current number of active sessions. |
191| haproxy_backend_max_sessions | Maximum observed number of active sessions. |
192| haproxy_backend_limit_sessions | Configured session limit. |
193| haproxy_backend_sessions_total | Total number of sessions. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100194| haproxy_backend_max_session_rate | Maximum observed number of sessions per second. |
195| haproxy_backend_last_session_seconds | Number of seconds since last session assigned to server/backend. |
196| haproxy_backend_current_queue | Current number of queued requests. |
197| haproxy_backend_max_queue | Maximum observed number of queued requests. |
198| haproxy_backend_connection_attempts_total | Total number of connection establishment attempts. |
199| haproxy_backend_connection_reuses_total | Total number of connection reuses. |
200| haproxy_backend_bytes_in_total | Current total of incoming bytes. |
201| haproxy_backend_bytes_out_total | Current total of outgoing bytes. |
Christopher Faulet68b69682019-11-08 15:12:29 +0100202| haproxy_backend_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. |
203| haproxy_backend_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. |
204| haproxy_backend_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) |
205| haproxy_backend_total_time_average_seconds | Avg. total time for last 1024 successful connections. |
Christopher Faulet8fc027d2019-11-08 15:05:31 +0100206| haproxy_backend_max_queue_time_seconds | Maximum observed queue time. |
207| haproxy_backend_max_connect_time_seconds | Maximum observed connect time. |
Christopher Faulet68b69682019-11-08 15:12:29 +0100208| haproxy_backend_max_response_time_seconds | Maximum observed response time. (0 for TCP) |
Christopher Faulet8fc027d2019-11-08 15:05:31 +0100209| haproxy_backend_max_total_time_seconds | Maximum observed total time. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100210| haproxy_backend_requests_denied_total | Total number of denied requests. |
211| haproxy_backend_responses_denied_total | Total number of denied responses. |
212| haproxy_backend_connection_errors_total | Total number of connection errors. |
213| haproxy_backend_response_errors_total | Total number of response errors. |
214| haproxy_backend_retry_warnings_total | Total number of retry warnings. |
215| haproxy_backend_redispatch_warnings_total | Total number of redispatch warnings. |
216| haproxy_backend_failed_header_rewriting_total | Total number of failed header rewriting warnings. |
217| haproxy_backend_client_aborts_total | Total number of data transfers aborted by the client. |
218| haproxy_backend_server_aborts_total | Total number of data transfers aborted by the server. |
219| haproxy_backend_weight | Service weight. |
220| haproxy_backend_active_servers | Current number of active servers. |
221| haproxy_backend_backup_servers | Current number of backup servers. |
222| haproxy_backend_check_up_down_total | Total number of UP->DOWN transitions. |
223| haproxy_backend_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. |
224| haproxy_backend_downtime_seconds_total | Total downtime (in seconds) for the service. |
225| haproxy_backend_loadbalanced_total | Total number of times a service was selected. |
226| haproxy_backend_http_requests_total | Total number of HTTP requests received. |
227| haproxy_backend_http_responses_total | Total number of HTTP responses. |
228| haproxy_backend_http_cache_lookups_total | Total number of HTTP cache lookups. |
229| haproxy_backend_http_cache_hits_total | Total number of HTTP cache hits. |
230| haproxy_backend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. |
231| haproxy_backend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. |
232| haproxy_backend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). |
233| haproxy_backend_http_comp_responses_total | Total number of HTTP responses that were compressed. |
Christopher Faulete4a2c8d2019-12-16 14:44:01 +0100234| haproxy_backend_internal_errors_total | Total number of internal errors. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100235+-----------------------------------------------------+--------------------------------------------------------------------------+
236
237* Server metrics
238
239+----------------------------------------------------+---------------------------------------------------------------------------+
240| Metric name | Description |
241+----------------------------------------------------+---------------------------------------------------------------------------+
242| haproxy_server_status | Current status of the service. |
243| haproxy_server_current_sessions | Current number of active sessions. |
244| haproxy_server_max_sessions | Maximum observed number of active sessions. |
245| haproxy_server_limit_sessions | Configured session limit. |
246| haproxy_server_sessions_total | Total number of sessions. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100247| haproxy_server_max_session_rate | Maximum observed number of sessions per second. |
248| haproxy_server_last_session_seconds | Number of seconds since last session assigned to server/backend. |
249| haproxy_server_current_queue | Current number of queued requests. |
250| haproxy_server_max_queue | Maximum observed number of queued requests. |
251| haproxy_server_queue_limit | Configured maxqueue for the server (0 meaning no limit). |
252| haproxy_server_bytes_in_total | Current total of incoming bytes. |
253| haproxy_server_bytes_out_total | Current total of outgoing bytes. |
Christopher Faulet68b69682019-11-08 15:12:29 +0100254| haproxy_server_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. |
255| haproxy_server_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. |
256| haproxy_server_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) |
257| haproxy_server_total_time_average_seconds | Avg. total time for last 1024 successful connections. |
Christopher Faulet8fc027d2019-11-08 15:05:31 +0100258| haproxy_server_max_queue_time_seconds | Maximum observed queue time. |
259| haproxy_server_max_connect_time_seconds | Maximum observed connect time. |
Christopher Faulet68b69682019-11-08 15:12:29 +0100260| haproxy_server_max_response_time_seconds | Maximum observed response time. (0 for TCP) |
Christopher Faulet8fc027d2019-11-08 15:05:31 +0100261| haproxy_server_max_total_time_seconds | Maximum observed total time. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100262| haproxy_server_connection_attempts_total | Total number of connection establishment attempts. |
263| haproxy_server_connection_reuses_total | Total number of connection reuses. |
264| haproxy_server_responses_denied_total | Total number of denied responses. |
265| haproxy_server_connection_errors_total | Total number of connection errors. |
266| haproxy_server_response_errors_total | Total number of response errors. |
267| haproxy_server_retry_warnings_total | Total number of retry warnings. |
268| haproxy_server_redispatch_warnings_total | Total number of redispatch warnings. |
269| haproxy_server_failed_header_rewriting_total | Total number of failed header rewriting warnings. |
270| haproxy_server_client_aborts_total | Total number of data transfers aborted by the client. |
271| haproxy_server_server_aborts_total | Total number of data transfers aborted by the server. |
272| haproxy_server_weight | Service weight. |
Christopher Fauletcf403f32019-11-21 14:35:46 +0100273| haproxy_server_check_status | Status of last health check, if enabled. (see below for the mapping) |
274| haproxy_server_check_code | layer5-7 code, if available of the last health check. |
Christopher Faulet2711e512020-02-27 16:12:07 +0100275| haproxy_server_check_duration_seconds | Total duration of the latest server health check, in seconds. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100276| haproxy_server_check_failures_total | Total number of failed check (Only when the server is up). |
277| haproxy_server_check_up_down_total | Total number of UP->DOWN transitions. |
278| haproxy_server_downtime_seconds_total | Total downtime (in seconds) for the service. |
279| haproxy_server_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. |
280| haproxy_server_current_throttle | Current throttle percentage for the server, when slowstart is active. |
281| haproxy_server_loadbalanced_total | Total number of times a service was selected. |
282| haproxy_server_http_responses_total | Total number of HTTP responses. |
Christopher Faulet20ab80c2019-11-08 15:24:32 +0100283| haproxy_server_idle_connections_current | Current number of idle connections available for reuse. |
284| haproxy_server_idle_connections_limit | Limit on the number of available idle connections. |
Christopher Faulete4a2c8d2019-12-16 14:44:01 +0100285| haproxy_server_internal_errors_total | Total number of internal errors. |
Christopher Fauletf959d082019-02-07 15:38:42 +0100286+----------------------------------------------------+---------------------------------------------------------------------------+
Christopher Fauletcf403f32019-11-21 14:35:46 +0100287
288Mapping of health check status :
289
290 0 : HCHK_STATUS_UNKNOWN (Unknown)
291 1 : HCHK_STATUS_INI (Initializing)
292
293 4 : HCHK_STATUS_HANA (Health analyze detected enough consecutive errors)
294
295 5 : HCHK_STATUS_SOCKERR (Socket error)
296
297 6 : HCHK_STATUS_L4OK (L4 check passed, for example tcp connect)
298 7 : HCHK_STATUS_L4TOUT (L4 timeout)
299 8 : HCHK_STATUS_L4CON (L4 connection problem)
300
301 9 : HCHK_STATUS_L6OK (L6 check passed)
302 10 : HCHK_STATUS_L6TOUT (L6 (SSL) timeout)
303 11 : HCHK_STATUS_L6RSP (L6 invalid response - protocol error)
304
305 12 : HCHK_STATUS_L7TOUT (L7 (HTTP/SMTP) timeout)
306 13 : HCHK_STATUS_L7RSP (L7 invalid response - protocol error)
307 15 : HCHK_STATUS_L7OKD (L7 check passed)
308 16 : HCHK_STATUS_L7OKCD (L7 check conditionally passed)
309 17 : HCHK_STATUS_L7STS (L7 response error, for example HTTP 5xx)
310
311 18 : HCHK_STATUS_PROCERR (External process check failure)
312 19 : HCHK_STATUS_PROCTOUT (External process check timeout)
313 20 : HCHK_STATUS_PROCOK (External process check passed)