blob: 4e29e23cab4c046adeceb4703c8749d2ca4b37b2 [file] [log] [blame]
Christopher Fauletf959d082019-02-07 15:38:42 +01001PROMEX: A Prometheus exporter for HAProxy
2-------------------------------------------
3
4Prometheus is a monitoring and alerting system. More and more people use it to
5monitor their environment (this is written February 2019). It collects metrics
6from monitored targets by scraping metrics HTTP endpoints on these targets. For
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +05007HAProxy, The Prometheus team officially supports an exporter written in Go
Christopher Fauletf959d082019-02-07 15:38:42 +01008(https://github.com/prometheus/haproxy_exporter). But it requires an extra
9software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
10exporter for HAProxy. It was developed as a service and is directly available in
11HAProxy, like the stats applet.
12
13However, PROMEX is not built by default with HAProxy. It is provided as an extra
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +050014component for everyone want to use it. So you need to explicitly build HAProxy
Willy Tarreau92dc7862021-04-02 15:31:10 +020015with the PROMEX service, setting the Makefile variable "USE_PROMEX" to "1". For
16instance:
Christopher Fauletf959d082019-02-07 15:38:42 +010017
Willy Tarreau92dc7862021-04-02 15:31:10 +020018 > make TARGET=linux-glibc USE_PROMEX=1
Christopher Fauletf959d082019-02-07 15:38:42 +010019
20if HAProxy provides the PROMEX service, the following build option will be
21reported by the command "haproxy -vv":
22
23 Built with the Prometheus exporter as a service
24
25To be used, it must be enabled in the configuration with an "http-request" rule
26and the corresponding HTTP proxy must enable the HTX support. For instance:
27
28 frontend test
29 mode http
30 ...
Christopher Fauletf959d082019-02-07 15:38:42 +010031 http-request use-service prometheus-exporter if { path /metrics }
32 ...
33
34
35This service has been developed as a third-party component because it could
36become obsolete, depending on how much time Prometheus will remain heavily
37used. This is said with no ulterior motive of course. Prometheus is a great
38software and I hope all the well for it. But we involve in a environment moving
39quickly and a solution may be obvious today could be deprecated the next
40year. And because PROMEX is not integrated by default into the HAProxy codebase,
41it will need some interest to be actively supported. All contribution of any
42kind are welcome.
43
44You must also be careful if you use with huge configurations. Unlike the stats
45applet, all metrics are not grouped by service (proxy, listener or server). With
46PROMEX, all lines for a given metric are provided as one single group. So
47instead of collecting all metrics for a proxy before moving to the next one, we
48must loop on all proxies for each metric. Same for the servers. Thus, it will
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +050049spend much more resources to produce the Prometheus metrics than the CSV export
Christopher Fauletf959d082019-02-07 15:38:42 +010050through the stats page. To give a comparison order, quick benchmarks shown that
51a PROMEX dump is 5x slower and 20x more verbose than a CSV export.
52
Christopher Faulet78407ce2019-11-18 14:47:08 +010053
54metrics filtering
55-------------------
56
57It is possible to dynamically select the metrics to export if you don't use all
58of them passing parameters in the query-string.
59
60* Filtering on scopes
61
62The metrics may be filtered by scopes. Multiple parameters with "scope" as name
63may be passed in the query-string to filter exported metrics, with one of those
64values: global, frontend, backend, server or '*' (means all). A scope parameter
65with no value means to filter out all scopes (nothing is returned). The scope
66parameters are parsed in their appearance order in the query-string. So an empty
67scope will reset all scopes already parsed. But it can be overridden by
68following scope parameters in the query-string. By default everything is
69exported. Here are examples:
70
71 /metrics?scope=server # ==> server metrics will be exported
72 /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
William Dauchye3f7bd52021-02-14 23:22:56 +010073 /metrics?scope=listener # ==> listener metrics will be exported
Christopher Faulet78407ce2019-11-18 14:47:08 +010074 /metrics?scope=*&scope= # ==> no metrics will be exported
75 /metrics?scope=&scope=global # ==> global metrics will be exported
William Dauchy69164222021-02-07 20:42:38 +010076 /metrics?scope=sticktable # ==> stick tables metrics will be exported
Christopher Faulet78407ce2019-11-18 14:47:08 +010077
William Dauchyde3c3262021-02-01 13:11:51 +010078* How do I prevent my prometheus instance to explode?
79
80** Filtering on servers state
Christopher Fauleteba22942019-11-19 14:18:24 +010081
82It is possible to exclude from returned metrics all servers in maintenance mode
83passing the parameter "no-maint" in the query-string. This parameter may help to
84solve performance issues of configuration that use the server templates to
85manage dynamic provisionning. Note there is no consistency check on the servers
86state. So, if the state of a server changes while the exporter is running, only
87a part of the metrics for this server will be dumped.
Christopher Faulet78407ce2019-11-18 14:47:08 +010088
William Dauchyde3c3262021-02-01 13:11:51 +010089prometheus example config:
90
91For server-template users:
92- <job>
93 params:
94 no-maint:
95 - empty
96
97** Scrap server health checks only
98
99All health checks status are dump through `state` label values. If you want to
100scrap server health check status but prevent all server metrics to be saved,
101except the server_check_status, you may configure prometheus that way:
102
103- <job>
104 metric_relabel_configs:
105 - source_labels: ['__name__']
William Dauchye3f7bd52021-02-14 23:22:56 +0100106 regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
William Dauchyde3c3262021-02-01 13:11:51 +0100107 action: keep
108
Christopher Fauletf959d082019-02-07 15:38:42 +0100109Exported metrics
110------------------
111
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100112See prometheus export for the description of each field.
113
Christopher Fauletf959d082019-02-07 15:38:42 +0100114* Globals metrics
115
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100116+------------------------------------------------+
117| Metric name |
118+------------------------------------------------+
119| haproxy_process_nbthread |
120| haproxy_process_nbproc |
121| haproxy_process_relative_process_id |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100122| haproxy_process_uptime_seconds |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100123| haproxy_process_pool_failures_total |
124| haproxy_process_max_fds |
125| haproxy_process_max_sockets |
126| haproxy_process_max_connections |
127| haproxy_process_hard_max_connections |
128| haproxy_process_current_connections |
129| haproxy_process_connections_total |
130| haproxy_process_requests_total |
131| haproxy_process_max_ssl_connections |
132| haproxy_process_current_ssl_connections |
133| haproxy_process_ssl_connections_total |
134| haproxy_process_max_pipes |
135| haproxy_process_pipes_used_total |
136| haproxy_process_pipes_free_total |
137| haproxy_process_current_connection_rate |
138| haproxy_process_limit_connection_rate |
139| haproxy_process_max_connection_rate |
140| haproxy_process_current_session_rate |
141| haproxy_process_limit_session_rate |
142| haproxy_process_max_session_rate |
143| haproxy_process_current_ssl_rate |
144| haproxy_process_limit_ssl_rate |
145| haproxy_process_max_ssl_rate |
146| haproxy_process_current_frontend_ssl_key_rate |
147| haproxy_process_max_frontend_ssl_key_rate |
148| haproxy_process_frontend_ssl_reuse |
149| haproxy_process_current_backend_ssl_key_rate |
150| haproxy_process_max_backend_ssl_key_rate |
151| haproxy_process_ssl_cache_lookups_total |
152| haproxy_process_ssl_cache_misses_total |
153| haproxy_process_http_comp_bytes_in_total |
154| haproxy_process_http_comp_bytes_out_total |
155| haproxy_process_limit_http_comp |
156| haproxy_process_current_zlib_memory |
157| haproxy_process_max_zlib_memory |
158| haproxy_process_current_tasks |
159| haproxy_process_current_run_queue |
160| haproxy_process_idle_time_percent |
161| haproxy_process_stopping |
162| haproxy_process_jobs |
163| haproxy_process_unstoppable_jobs |
164| haproxy_process_listeners |
165| haproxy_process_active_peers |
166| haproxy_process_connected_peers |
167| haproxy_process_dropped_logs_total |
168| haproxy_process_busy_polling_enabled |
169| haproxy_process_failed_resolutions |
170| haproxy_process_bytes_out_total |
171| haproxy_process_spliced_bytes_out_total |
172| haproxy_process_bytes_out_rate |
173| haproxy_process_recv_logs_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100174| haproxy_process_build_info |
175| haproxy_process_max_memory_bytes |
176| haproxy_process_pool_allocated_bytes |
177| haproxy_process_pool_used_bytes |
178| haproxy_process_start_time_seconds |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100179+------------------------------------------------+
Christopher Fauletf959d082019-02-07 15:38:42 +0100180
181* Frontend metrics
182
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100183+-------------------------------------------------+
184| Metric name |
185+-------------------------------------------------+
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100186| haproxy_frontend_current_sessions |
187| haproxy_frontend_max_sessions |
188| haproxy_frontend_limit_sessions |
189| haproxy_frontend_sessions_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100190| haproxy_frontend_bytes_in_total |
191| haproxy_frontend_bytes_out_total |
192| haproxy_frontend_requests_denied_total |
193| haproxy_frontend_responses_denied_total |
194| haproxy_frontend_request_errors_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100195| haproxy_frontend_status |
196| haproxy_frontend_limit_session_rate |
197| haproxy_frontend_max_session_rate |
198| haproxy_frontend_http_responses_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100199| haproxy_frontend_http_requests_rate_max |
200| haproxy_frontend_http_requests_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100201| haproxy_frontend_http_comp_bytes_in_total |
202| haproxy_frontend_http_comp_bytes_out_total |
203| haproxy_frontend_http_comp_bytes_bypassed_total |
204| haproxy_frontend_http_comp_responses_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100205| haproxy_frontend_connections_rate_max |
206| haproxy_frontend_connections_total |
207| haproxy_frontend_intercepted_requests_total |
208| haproxy_frontend_denied_connections_total |
209| haproxy_frontend_denied_sessions_total |
210| haproxy_frontend_failed_header_rewriting_total |
211| haproxy_frontend_http_cache_lookups_total |
212| haproxy_frontend_http_cache_hits_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100213| haproxy_frontend_internal_errors_total |
214+-------------------------------------------------+
Christopher Fauletf959d082019-02-07 15:38:42 +0100215
William Dauchye3f7bd52021-02-14 23:22:56 +0100216* Listener metrics
217
218+-------------------------------------------------+
219| Metric name |
220+-------------------------------------------------+
221| haproxy_listener_current_sessions |
222| haproxy_listener_max_sessions |
223| haproxy_listener_limit_sessions |
224| haproxy_listener_sessions_total |
225| haproxy_listener_bytes_in_total |
226| haproxy_listener_bytes_out_total |
227| haproxy_listener_requests_denied_total |
228| haproxy_listener_responses_denied_total |
229| haproxy_listener_request_errors_total |
230| haproxy_listener_status |
231| haproxy_listener_denied_connections_total |
232| haproxy_listener_denied_sessions_total |
233| haproxy_listener_failed_header_rewriting_total |
234| haproxy_listener_internal_errors_total |
235+-------------------------------------------------+
236
Rick Rackow35efbe22019-10-08 11:28:06 +0200237* Backend metrics
Christopher Fauletf959d082019-02-07 15:38:42 +0100238
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100239+-----------------------------------------------------+
240| Metric name |
241+-----------------------------------------------------+
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100242| haproxy_backend_current_queue |
243| haproxy_backend_max_queue |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100244| haproxy_backend_current_sessions |
245| haproxy_backend_max_sessions |
246| haproxy_backend_limit_sessions |
247| haproxy_backend_sessions_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100248| haproxy_backend_bytes_in_total |
249| haproxy_backend_bytes_out_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100250| haproxy_backend_requests_denied_total |
251| haproxy_backend_responses_denied_total |
252| haproxy_backend_connection_errors_total |
253| haproxy_backend_response_errors_total |
254| haproxy_backend_retry_warnings_total |
255| haproxy_backend_redispatch_warnings_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100256| haproxy_backend_status |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100257| haproxy_backend_weight |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100258| haproxy_backend_active_servers |
259| haproxy_backend_backup_servers |
260| haproxy_backend_check_up_down_total |
261| haproxy_backend_check_last_change_seconds |
262| haproxy_backend_downtime_seconds_total |
263| haproxy_backend_loadbalanced_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100264| haproxy_backend_max_session_rate |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100265| haproxy_backend_http_responses_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100266| haproxy_backend_http_requests_total |
267| haproxy_backend_client_aborts_total |
268| haproxy_backend_server_aborts_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100269| haproxy_backend_http_comp_bytes_in_total |
270| haproxy_backend_http_comp_bytes_out_total |
271| haproxy_backend_http_comp_bytes_bypassed_total |
272| haproxy_backend_http_comp_responses_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100273| haproxy_backend_last_session_seconds |
274| haproxy_backend_queue_time_average_seconds |
275| haproxy_backend_connect_time_average_seconds |
276| haproxy_backend_response_time_average_seconds |
277| haproxy_backend_total_time_average_seconds |
278| haproxy_backend_failed_header_rewriting_total |
279| haproxy_backend_connection_attempts_total |
280| haproxy_backend_connection_reuses_total |
281| haproxy_backend_http_cache_lookups_total |
282| haproxy_backend_http_cache_hits_total |
283| haproxy_backend_max_queue_time_seconds |
284| haproxy_backend_max_connect_time_seconds |
285| haproxy_backend_max_response_time_seconds |
286| haproxy_backend_max_total_time_seconds |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100287| haproxy_backend_internal_errors_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100288| haproxy_backend_uweight |
Christopher Faulet82e495c2022-12-09 11:04:11 +0100289| haproxy_backend_agg_server_status |
290| haproxy_backend_agg_check_status |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100291+-----------------------------------------------------+
Christopher Fauletf959d082019-02-07 15:38:42 +0100292
293* Server metrics
294
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100295+----------------------------------------------------+
296| Metric name |
297+----------------------------------------------------+
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100298| haproxy_server_current_queue |
299| haproxy_server_max_queue |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100300| haproxy_server_current_sessions |
301| haproxy_server_max_sessions |
302| haproxy_server_limit_sessions |
303| haproxy_server_sessions_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100304| haproxy_server_bytes_in_total |
305| haproxy_server_bytes_out_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100306| haproxy_server_responses_denied_total |
307| haproxy_server_connection_errors_total |
308| haproxy_server_response_errors_total |
309| haproxy_server_retry_warnings_total |
310| haproxy_server_redispatch_warnings_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100311| haproxy_server_status |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100312| haproxy_server_weight |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100313| haproxy_server_check_failures_total |
314| haproxy_server_check_up_down_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100315| haproxy_server_check_last_change_seconds |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100316| haproxy_server_downtime_seconds_total |
317| haproxy_server_queue_limit |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100318| haproxy_server_current_throttle |
319| haproxy_server_loadbalanced_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100320| haproxy_server_max_session_rate |
321| haproxy_server_check_status |
322| haproxy_server_check_code |
323| haproxy_server_check_duration_seconds |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100324| haproxy_server_http_responses_total |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100325| haproxy_server_client_aborts_total |
326| haproxy_server_server_aborts_total |
327| haproxy_server_last_session_seconds |
328| haproxy_server_queue_time_average_seconds |
329| haproxy_server_connect_time_average_seconds |
330| haproxy_server_response_time_average_seconds |
331| haproxy_server_total_time_average_seconds |
332| haproxy_server_failed_header_rewriting_total |
333| haproxy_server_connection_attempts_total |
334| haproxy_server_connection_reuses_total |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100335| haproxy_server_idle_connections_current |
336| haproxy_server_idle_connections_limit |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100337| haproxy_server_max_queue_time_seconds |
338| haproxy_server_max_connect_time_seconds |
339| haproxy_server_max_response_time_seconds |
340| haproxy_server_max_total_time_seconds |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100341| haproxy_server_internal_errors_total |
342| haproxy_server_unsafe_idle_connections_current |
343| haproxy_server_safe_idle_connections_current |
344| haproxy_server_used_connections_current |
345| haproxy_server_need_connections_current |
Christopher Faulet1a68cd02021-02-01 14:50:30 +0100346| haproxy_server_uweight |
William Dauchy4b7bf7e2021-02-01 13:12:00 +0100347+----------------------------------------------------+
William Dauchy69164222021-02-07 20:42:38 +0100348
349* Stick table metrics
350
351+----------------------------------------------------+
352| Metric name |
353+----------------------------------------------------+
354| haproxy_sticktable_size |
355| haproxy_sticktable_used |
356+----------------------------------------------------+