| PROMEX: A Prometheus exporter for HAProxy |
| ------------------------------------------- |
| |
| Prometheus is a monitoring and alerting system. More and more people use it to |
| monitor their environment (this is written February 2019). It collects metrics |
| from monitored targets by scraping metrics HTTP endpoints on these targets. For |
| HAProxy, The Prometheus team officially supports an exporter written in Go |
| (https://github.com/prometheus/haproxy_exporter). But it requires an extra |
| software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus |
| exporter for HAProxy. It was developed as a service and is directly available in |
| HAProxy, like the stats applet. |
| |
| However, PROMEX is not built by default with HAProxy. It is provided as an extra |
| component for everyone want to use it. So you need to explicitly build HAProxy |
| with the PROMEX service, setting the Makefile variable "USE_PROMEX" to "1". For |
| instance: |
| |
| > make TARGET=linux-glibc USE_PROMEX=1 |
| |
| if HAProxy provides the PROMEX service, the following build option will be |
| reported by the command "haproxy -vv": |
| |
| Built with the Prometheus exporter as a service |
| |
| To be used, it must be enabled in the configuration with an "http-request" rule |
| and the corresponding HTTP proxy must enable the HTX support. For instance: |
| |
| frontend test |
| mode http |
| ... |
| http-request use-service prometheus-exporter if { path /metrics } |
| ... |
| |
| |
| This service has been developed as a third-party component because it could |
| become obsolete, depending on how much time Prometheus will remain heavily |
| used. This is said with no ulterior motive of course. Prometheus is a great |
| software and I hope all the well for it. But we involve in a environment moving |
| quickly and a solution may be obvious today could be deprecated the next |
| year. And because PROMEX is not integrated by default into the HAProxy codebase, |
| it will need some interest to be actively supported. All contribution of any |
| kind are welcome. |
| |
| You must also be careful if you use with huge configurations. Unlike the stats |
| applet, all metrics are not grouped by service (proxy, listener or server). With |
| PROMEX, all lines for a given metric are provided as one single group. So |
| instead of collecting all metrics for a proxy before moving to the next one, we |
| must loop on all proxies for each metric. Same for the servers. Thus, it will |
| spend much more resources to produce the Prometheus metrics than the CSV export |
| through the stats page. To give a comparison order, quick benchmarks shown that |
| a PROMEX dump is 5x slower and 20x more verbose than a CSV export. |
| |
| |
| metrics filtering |
| ------------------- |
| |
| It is possible to dynamically select the metrics to export if you don't use all |
| of them passing parameters in the query-string. |
| |
| * Filtering on scopes |
| |
| The metrics may be filtered by scopes. Multiple parameters with "scope" as name |
| may be passed in the query-string to filter exported metrics, with one of those |
| values: global, frontend, backend, server or '*' (means all). A scope parameter |
| with no value means to filter out all scopes (nothing is returned). The scope |
| parameters are parsed in their appearance order in the query-string. So an empty |
| scope will reset all scopes already parsed. But it can be overridden by |
| following scope parameters in the query-string. By default everything is |
| exported. Here are examples: |
| |
| /metrics?scope=server # ==> server metrics will be exported |
| /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported |
| /metrics?scope=listener # ==> listener metrics will be exported |
| /metrics?scope=*&scope= # ==> no metrics will be exported |
| /metrics?scope=&scope=global # ==> global metrics will be exported |
| /metrics?scope=sticktable # ==> stick tables metrics will be exported |
| |
| * How do I prevent my prometheus instance to explode? |
| |
| ** Filtering on servers state |
| |
| It is possible to exclude from returned metrics all servers in maintenance mode |
| passing the parameter "no-maint" in the query-string. This parameter may help to |
| solve performance issues of configuration that use the server templates to |
| manage dynamic provisionning. Note there is no consistency check on the servers |
| state. So, if the state of a server changes while the exporter is running, only |
| a part of the metrics for this server will be dumped. |
| |
| prometheus example config: |
| |
| For server-template users: |
| - <job> |
| params: |
| no-maint: |
| - empty |
| |
| ** Scrap server health checks only |
| |
| All health checks status are dump through `state` label values. If you want to |
| scrap server health check status but prevent all server metrics to be saved, |
| except the server_check_status, you may configure prometheus that way: |
| |
| - <job> |
| metric_relabel_configs: |
| - source_labels: ['__name__'] |
| regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*' |
| action: keep |
| |
| Exported metrics |
| ------------------ |
| |
| See prometheus export for the description of each field. |
| |
| * Globals metrics |
| |
| +------------------------------------------------+ |
| | Metric name | |
| +------------------------------------------------+ |
| | haproxy_process_nbthread | |
| | haproxy_process_nbproc | |
| | haproxy_process_relative_process_id | |
| | haproxy_process_uptime_seconds | |
| | haproxy_process_pool_failures_total | |
| | haproxy_process_max_fds | |
| | haproxy_process_max_sockets | |
| | haproxy_process_max_connections | |
| | haproxy_process_hard_max_connections | |
| | haproxy_process_current_connections | |
| | haproxy_process_connections_total | |
| | haproxy_process_requests_total | |
| | haproxy_process_max_ssl_connections | |
| | haproxy_process_current_ssl_connections | |
| | haproxy_process_ssl_connections_total | |
| | haproxy_process_max_pipes | |
| | haproxy_process_pipes_used_total | |
| | haproxy_process_pipes_free_total | |
| | haproxy_process_current_connection_rate | |
| | haproxy_process_limit_connection_rate | |
| | haproxy_process_max_connection_rate | |
| | haproxy_process_current_session_rate | |
| | haproxy_process_limit_session_rate | |
| | haproxy_process_max_session_rate | |
| | haproxy_process_current_ssl_rate | |
| | haproxy_process_limit_ssl_rate | |
| | haproxy_process_max_ssl_rate | |
| | haproxy_process_current_frontend_ssl_key_rate | |
| | haproxy_process_max_frontend_ssl_key_rate | |
| | haproxy_process_frontend_ssl_reuse | |
| | haproxy_process_current_backend_ssl_key_rate | |
| | haproxy_process_max_backend_ssl_key_rate | |
| | haproxy_process_ssl_cache_lookups_total | |
| | haproxy_process_ssl_cache_misses_total | |
| | haproxy_process_http_comp_bytes_in_total | |
| | haproxy_process_http_comp_bytes_out_total | |
| | haproxy_process_limit_http_comp | |
| | haproxy_process_current_zlib_memory | |
| | haproxy_process_max_zlib_memory | |
| | haproxy_process_current_tasks | |
| | haproxy_process_current_run_queue | |
| | haproxy_process_idle_time_percent | |
| | haproxy_process_stopping | |
| | haproxy_process_jobs | |
| | haproxy_process_unstoppable_jobs | |
| | haproxy_process_listeners | |
| | haproxy_process_active_peers | |
| | haproxy_process_connected_peers | |
| | haproxy_process_dropped_logs_total | |
| | haproxy_process_busy_polling_enabled | |
| | haproxy_process_failed_resolutions | |
| | haproxy_process_bytes_out_total | |
| | haproxy_process_spliced_bytes_out_total | |
| | haproxy_process_bytes_out_rate | |
| | haproxy_process_recv_logs_total | |
| | haproxy_process_build_info | |
| | haproxy_process_max_memory_bytes | |
| | haproxy_process_pool_allocated_bytes | |
| | haproxy_process_pool_used_bytes | |
| | haproxy_process_start_time_seconds | |
| +------------------------------------------------+ |
| |
| * Frontend metrics |
| |
| +-------------------------------------------------+ |
| | Metric name | |
| +-------------------------------------------------+ |
| | haproxy_frontend_current_sessions | |
| | haproxy_frontend_max_sessions | |
| | haproxy_frontend_limit_sessions | |
| | haproxy_frontend_sessions_total | |
| | haproxy_frontend_bytes_in_total | |
| | haproxy_frontend_bytes_out_total | |
| | haproxy_frontend_requests_denied_total | |
| | haproxy_frontend_responses_denied_total | |
| | haproxy_frontend_request_errors_total | |
| | haproxy_frontend_status | |
| | haproxy_frontend_limit_session_rate | |
| | haproxy_frontend_max_session_rate | |
| | haproxy_frontend_http_responses_total | |
| | haproxy_frontend_http_requests_rate_max | |
| | haproxy_frontend_http_requests_total | |
| | haproxy_frontend_http_comp_bytes_in_total | |
| | haproxy_frontend_http_comp_bytes_out_total | |
| | haproxy_frontend_http_comp_bytes_bypassed_total | |
| | haproxy_frontend_http_comp_responses_total | |
| | haproxy_frontend_connections_rate_max | |
| | haproxy_frontend_connections_total | |
| | haproxy_frontend_intercepted_requests_total | |
| | haproxy_frontend_denied_connections_total | |
| | haproxy_frontend_denied_sessions_total | |
| | haproxy_frontend_failed_header_rewriting_total | |
| | haproxy_frontend_http_cache_lookups_total | |
| | haproxy_frontend_http_cache_hits_total | |
| | haproxy_frontend_internal_errors_total | |
| +-------------------------------------------------+ |
| |
| * Listener metrics |
| |
| +-------------------------------------------------+ |
| | Metric name | |
| +-------------------------------------------------+ |
| | haproxy_listener_current_sessions | |
| | haproxy_listener_max_sessions | |
| | haproxy_listener_limit_sessions | |
| | haproxy_listener_sessions_total | |
| | haproxy_listener_bytes_in_total | |
| | haproxy_listener_bytes_out_total | |
| | haproxy_listener_requests_denied_total | |
| | haproxy_listener_responses_denied_total | |
| | haproxy_listener_request_errors_total | |
| | haproxy_listener_status | |
| | haproxy_listener_denied_connections_total | |
| | haproxy_listener_denied_sessions_total | |
| | haproxy_listener_failed_header_rewriting_total | |
| | haproxy_listener_internal_errors_total | |
| +-------------------------------------------------+ |
| |
| * Backend metrics |
| |
| +-----------------------------------------------------+ |
| | Metric name | |
| +-----------------------------------------------------+ |
| | haproxy_backend_current_queue | |
| | haproxy_backend_max_queue | |
| | haproxy_backend_current_sessions | |
| | haproxy_backend_max_sessions | |
| | haproxy_backend_limit_sessions | |
| | haproxy_backend_sessions_total | |
| | haproxy_backend_bytes_in_total | |
| | haproxy_backend_bytes_out_total | |
| | haproxy_backend_requests_denied_total | |
| | haproxy_backend_responses_denied_total | |
| | haproxy_backend_connection_errors_total | |
| | haproxy_backend_response_errors_total | |
| | haproxy_backend_retry_warnings_total | |
| | haproxy_backend_redispatch_warnings_total | |
| | haproxy_backend_status | |
| | haproxy_backend_weight | |
| | haproxy_backend_active_servers | |
| | haproxy_backend_backup_servers | |
| | haproxy_backend_check_up_down_total | |
| | haproxy_backend_check_last_change_seconds | |
| | haproxy_backend_downtime_seconds_total | |
| | haproxy_backend_loadbalanced_total | |
| | haproxy_backend_max_session_rate | |
| | haproxy_backend_http_responses_total | |
| | haproxy_backend_http_requests_total | |
| | haproxy_backend_client_aborts_total | |
| | haproxy_backend_server_aborts_total | |
| | haproxy_backend_http_comp_bytes_in_total | |
| | haproxy_backend_http_comp_bytes_out_total | |
| | haproxy_backend_http_comp_bytes_bypassed_total | |
| | haproxy_backend_http_comp_responses_total | |
| | haproxy_backend_last_session_seconds | |
| | haproxy_backend_queue_time_average_seconds | |
| | haproxy_backend_connect_time_average_seconds | |
| | haproxy_backend_response_time_average_seconds | |
| | haproxy_backend_total_time_average_seconds | |
| | haproxy_backend_failed_header_rewriting_total | |
| | haproxy_backend_connection_attempts_total | |
| | haproxy_backend_connection_reuses_total | |
| | haproxy_backend_http_cache_lookups_total | |
| | haproxy_backend_http_cache_hits_total | |
| | haproxy_backend_max_queue_time_seconds | |
| | haproxy_backend_max_connect_time_seconds | |
| | haproxy_backend_max_response_time_seconds | |
| | haproxy_backend_max_total_time_seconds | |
| | haproxy_backend_internal_errors_total | |
| | haproxy_backend_uweight | |
| +-----------------------------------------------------+ |
| |
| * Server metrics |
| |
| +----------------------------------------------------+ |
| | Metric name | |
| +----------------------------------------------------+ |
| | haproxy_server_current_queue | |
| | haproxy_server_max_queue | |
| | haproxy_server_current_sessions | |
| | haproxy_server_max_sessions | |
| | haproxy_server_limit_sessions | |
| | haproxy_server_sessions_total | |
| | haproxy_server_bytes_in_total | |
| | haproxy_server_bytes_out_total | |
| | haproxy_server_responses_denied_total | |
| | haproxy_server_connection_errors_total | |
| | haproxy_server_response_errors_total | |
| | haproxy_server_retry_warnings_total | |
| | haproxy_server_redispatch_warnings_total | |
| | haproxy_server_status | |
| | haproxy_server_weight | |
| | haproxy_server_check_failures_total | |
| | haproxy_server_check_up_down_total | |
| | haproxy_server_check_last_change_seconds | |
| | haproxy_server_downtime_seconds_total | |
| | haproxy_server_queue_limit | |
| | haproxy_server_current_throttle | |
| | haproxy_server_loadbalanced_total | |
| | haproxy_server_max_session_rate | |
| | haproxy_server_check_status | |
| | haproxy_server_check_code | |
| | haproxy_server_check_duration_seconds | |
| | haproxy_server_http_responses_total | |
| | haproxy_server_client_aborts_total | |
| | haproxy_server_server_aborts_total | |
| | haproxy_server_last_session_seconds | |
| | haproxy_server_queue_time_average_seconds | |
| | haproxy_server_connect_time_average_seconds | |
| | haproxy_server_response_time_average_seconds | |
| | haproxy_server_total_time_average_seconds | |
| | haproxy_server_failed_header_rewriting_total | |
| | haproxy_server_connection_attempts_total | |
| | haproxy_server_connection_reuses_total | |
| | haproxy_server_idle_connections_current | |
| | haproxy_server_idle_connections_limit | |
| | haproxy_server_max_queue_time_seconds | |
| | haproxy_server_max_connect_time_seconds | |
| | haproxy_server_max_response_time_seconds | |
| | haproxy_server_max_total_time_seconds | |
| | haproxy_server_internal_errors_total | |
| | haproxy_server_unsafe_idle_connections_current | |
| | haproxy_server_safe_idle_connections_current | |
| | haproxy_server_used_connections_current | |
| | haproxy_server_need_connections_current | |
| | haproxy_server_uweight | |
| +----------------------------------------------------+ |
| |
| * Stick table metrics |
| |
| +----------------------------------------------------+ |
| | Metric name | |
| +----------------------------------------------------+ |
| | haproxy_sticktable_size | |
| | haproxy_sticktable_used | |
| +----------------------------------------------------+ |