PROMEX: A Prometheus exporter for HAProxy
-------------------------------------------

Prometheus is a monitoring and alerting system. More and more people use it to
monitor their environment (this is written February 2019). It collects metrics
from monitored targets by scraping metrics HTTP endpoints on these targets. For
HAProxy, The Prometheus team officially supports an exporter written in Go
(https://github.com/prometheus/haproxy_exporter). But it requires an extra
software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
exporter for HAProxy. It was developed as a service and is directly available in
HAProxy, like the stats applet.

However, PROMEX is not built by default with HAProxy. It is provided as an extra
component for everyone want to use it. So you need to explicitly build HAProxy
with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance:

    > make TARGET=linux-glibc EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

if HAProxy provides the PROMEX service, the following build option will be
reported by the command "haproxy -vv":

    Built with the Prometheus exporter as a service

To be used, it must be enabled in the configuration with an "http-request" rule
and the corresponding HTTP proxy must enable the HTX support. For instance:

    frontend test
        mode http
        ...
        option http-use-htx
        http-request use-service prometheus-exporter if { path /metrics }
        ...


This service has been developed as a third-party component because it could
become obsolete, depending on how much time Prometheus will remain heavily
used. This is said with no ulterior motive of course. Prometheus is a great
software and I hope all the well for it. But we involve in a environment moving
quickly and a solution may be obvious today could be deprecated the next
year. And because PROMEX is not integrated by default into the HAProxy codebase,
it will need some interest to be actively supported. All contribution of any
kind are welcome.

You must also be careful if you use with huge configurations. Unlike the stats
applet, all metrics are not grouped by service (proxy, listener or server). With
PROMEX, all lines for a given metric are provided as one single group. So
instead of collecting all metrics for a proxy before moving to the next one, we
must loop on all proxies for each metric. Same for the servers. Thus, it will
spend much more resources to produce the Prometheus metrics than the CSV export
through the stats page. To give a comparison order, quick benchmarks shown that
a PROMEX dump is 5x slower and 20x more verbose than a CSV export.


metrics filtering
-------------------

It is possible to dynamically select the metrics to export if you don't use all
of them passing parameters in the query-string.

* Filtering on scopes

The metrics may be filtered by scopes. Multiple parameters with "scope" as name
may be passed in the query-string to filter exported metrics, with one of those
values: global, frontend, backend, server or '*' (means all). A scope parameter
with no value means to filter out all scopes (nothing is returned). The scope
parameters are parsed in their appearance order in the query-string. So an empty
scope will reset all scopes already parsed. But it can be overridden by
following scope parameters in the query-string. By default everything is
exported. Here are examples:

  /metrics?scope=server                 # ==> server metrics will be exported
  /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
  /metrics?scope=*&scope=               # ==> no metrics will be exported
  /metrics?scope=&scope=global          # ==> global metrics will be exported

* How do I prevent my prometheus instance to explode?

** Filtering on servers state

It is possible to exclude from returned metrics all servers in maintenance mode
passing the parameter "no-maint" in the query-string. This parameter may help to
solve performance issues of configuration that use the server templates to
manage dynamic provisionning. Note there is no consistency check on the servers
state. So, if the state of a server changes while the exporter is running, only
a part of the metrics for this server will be dumped.

prometheus example config:

For server-template users:
- <job>
  params:
    no-maint:
    - empty

** Scrap server health checks only

All health checks status are dump through `state` label values. If you want to
scrap server health check status but prevent all server metrics to be saved,
except the server_check_status, you may configure prometheus that way:

- <job>
   metric_relabel_configs:
   - source_labels: ['__name__']
      regex: 'haproxy_(process_|frontend_|backend_|server_check_status).*'
      action: keep

Exported metrics
------------------

* Globals metrics

+------------------------------------------------+-------------------------------------------------------------------------------+
|    Metric name                                 |    Description                                                                |
+------------------------------------------------+-------------------------------------------------------------------------------+
| haproxy_process_nbthread                       | Configured number of threads.                                                 |
| haproxy_process_nbproc                         | Configured number of processes.                                               |
| haproxy_process_relative_process_id            | Relative process id, starting at 1.                                           |
| haproxy_process_start_time_seconds             | Start time in seconds.                                                        |
| haproxy_process_max_memory_bytes               | Per-process memory limit (in bytes); 0=unset.                                 |
| haproxy_process_pool_allocated_bytes           | Total amount of memory allocated in pools (in bytes).                         |
| haproxy_process_pool_used_bytes                | Total amount of memory used in pools (in bytes).                              |
| haproxy_process_pool_failures_total            | Total number of failed pool allocations.                                      |
| haproxy_process_max_fds                        | Maximum number of open file descriptors; 0=unset.                             |
| haproxy_process_max_sockets                    | Maximum number of open sockets.                                               |
| haproxy_process_max_connections                | Maximum number of concurrent connections.                                     |
| haproxy_process_hard_max_connections           | Initial Maximum number of concurrent connections.                             |
| haproxy_process_current_connections            | Number of active sessions.                                                    |
| haproxy_process_connections_total              | Total number of created sessions.                                             |
| haproxy_process_requests_total                 | Total number of requests (TCP or HTTP).                                       |
| haproxy_process_max_ssl_connections            | Configured maximum number of concurrent SSL connections.                      |
| haproxy_process_current_ssl_connections        | Number of opened SSL connections.                                             |
| haproxy_process_ssl_connections_total          | Total number of opened SSL connections.                                       |
| haproxy_process_max_pipes                      | Configured maximum number of pipes.                                           |
| haproxy_process_pipes_used_total               | Number of pipes in used.                                                      |
| haproxy_process_pipes_free_total               | Number of pipes unused.                                                       |
| haproxy_process_current_connection_rate        | Current number of connections per second over last elapsed second.            |
| haproxy_process_limit_connection_rate          | Configured maximum number of connections per second.                          |
| haproxy_process_max_connection_rate            | Maximum observed number of connections per second.                            |
| haproxy_process_current_session_rate           | Current number of sessions per second over last elapsed second.               |
| haproxy_process_limit_session_rate             | Configured maximum number of sessions per second.                             |
| haproxy_process_max_session_rate               | Maximum observed number of sessions per second.                               |
| haproxy_process_current_ssl_rate               | Current number of SSL sessions per second over last elapsed second.           |
| haproxy_process_limit_ssl_rate                 | Configured maximum number of SSL sessions per second.                         |
| haproxy_process_max_ssl_rate                   | Maximum observed number of SSL sessions per second.                           |
| haproxy_process_current_frontend_ssl_key_rate  | Current frontend SSL Key computation per second over last elapsed second.     |
| haproxy_process_max_frontend_ssl_key_rate      | Maximum observed frontend SSL Key computation per second.                     |
| haproxy_process_frontend_ssl_reuse             | SSL session reuse ratio (percent).                                            |
| haproxy_process_current_backend_ssl_key_rate   | Current backend SSL Key computation per second over last elapsed second.      |
| haproxy_process_max_backend_ssl_key_rate       | Maximum observed backend SSL Key computation per second.                      |
| haproxy_process_ssl_cache_lookups_total        | Total number of SSL session cache lookups.                                    |
| haproxy_process_ssl_cache_misses_total         | Total number of SSL session cache misses.                                     |
| haproxy_process_http_comp_bytes_in_total       | Number of bytes per second over last elapsed second, before http compression. |
| haproxy_process_http_comp_bytes_out_total      | Number of bytes per second over last elapsed second, after http compression.  |
| haproxy_process_limit_http_comp                | Configured maximum input compression rate in bytes.                           |
| haproxy_process_current_zlib_memory            | Current memory used for zlib in bytes.                                        |
| haproxy_process_max_zlib_memory                | Configured maximum amount of memory for zlib in bytes.                        |
| haproxy_process_current_tasks                  | Current number of tasks.                                                      |
| haproxy_process_current_run_queue              | Current number of tasks in the run-queue.                                     |
| haproxy_process_idle_time_percent              | Idle to total ratio over last sample (percent).                               |
| haproxy_process_stopping                       | Non zero means stopping in progress.                                          |
| haproxy_process_jobs                           | Current number of active jobs (listeners, sessions, open devices).            |
| haproxy_process_unstoppable_jobs               | Current number of active jobs that can't be stopped during a soft stop.       |
| haproxy_process_listeners                      | Current number of active listeners.                                           |
| haproxy_process_active_peers                   | Current number of active peers.                                               |
| haproxy_process_connected_peers                | Current number of connected peers.                                            |
| haproxy_process_dropped_logs_total             | Total number of dropped logs.                                                 |
| haproxy_process_busy_polling_enabled           | Non zero if the busy polling is enabled.                                      |
| haproxy_process_failed_resolutions             | Total number of failed DNS resolutions.                                       |
| haproxy_process_bytes_out_total                | Total number of bytes emitted.                                                |
| haproxy_process_spliced_bytes_out_total        | Total number of bytes emitted through a kernel pipe.                          |
| haproxy_process_bytes_out_rate                 | Number of bytes emitted over the last elapsed second.                         |
| haproxy_process_recv_logs_total                | Total number of log messages received by log-forwarding listeners on this     |
|                                                | worker process since started                                                  |
+------------------------------------------------+-------------------------------------------------------------------------------+

* Frontend metrics

+-------------------------------------------------+------------------------------------------------------------------------------+
|    Metric name                                  |    Description                                                               |
+-------------------------------------------------+------------------------------------------------------------------------------+
| haproxy_frontend_status                         | Current status of the service.                                               |
| haproxy_frontend_current_sessions               | Current number of active sessions.                                           |
| haproxy_frontend_max_sessions                   | Maximum observed number of active sessions.                                  |
| haproxy_frontend_limit_sessions                 | Configured session limit.                                                    |
| haproxy_frontend_sessions_total                 | Total number of sessions.                                                    |
| haproxy_frontend_limit_session_rate             | Configured limit on new sessions per second.                                 |
| haproxy_frontend_max_session_rate               | Maximum observed number of sessions per second.                              |
| haproxy_frontend_connections_rate_current       | Current number of connections per second over the last elapsed second.       |
| haproxy_frontend_connections_rate_max           | Maximum observed number of connections per second.                           |
| haproxy_frontend_connections_total              | Total number of connections.                                                 |
| haproxy_frontend_bytes_in_total                 | Current total of incoming bytes.                                             |
| haproxy_frontend_bytes_out_total                | Current total of outgoing bytes.                                             |
| haproxy_frontend_requests_denied_total          | Total number of denied requests.                                             |
| haproxy_frontend_responses_denied_total         | Total number of denied responses.                                            |
| haproxy_frontend_request_errors_total           | Total number of request errors.                                              |
| haproxy_frontend_denied_connections_total       | Total number of requests denied by "tcp-request connection" rules.           |
| haproxy_frontend_denied_sessions_total          | Total number of requests denied by "tcp-request session" rules.              |
| haproxy_frontend_failed_header_rewriting_total  | Total number of failed header rewriting warnings.                            |
| haproxy_frontend_http_requests_rate_max         | Maximum observed number of HTTP requests per second.                         |
| haproxy_frontend_http_requests_total            | Total number of HTTP requests received.                                      |
| haproxy_frontend_http_responses_total           | Total number of HTTP responses.                                              |
| haproxy_frontend_intercepted_requests_total     | Total number of intercepted HTTP requests.                                   |
| haproxy_frontend_http_cache_lookups_total       | Total number of HTTP cache lookups.                                          |
| haproxy_frontend_http_cache_hits_total          | Total number of HTTP cache hits.                                             |
| haproxy_frontend_http_comp_bytes_in_total       | Total number of HTTP response bytes fed to the compressor.                   |
| haproxy_frontend_http_comp_bytes_out_total      | Total number of HTTP response bytes emitted by the compressor.               |
| haproxy_frontend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit).      |
| haproxy_frontend_http_comp_responses_total      | Total number of HTTP responses that were compressed.                         |
| haproxy_frontend_internal_errors_total          | Total number of internal errors.                                             |
+-------------------------------------------------+------------------------------------------------------------------------------+

* Backend metrics

+-----------------------------------------------------+--------------------------------------------------------------------------+
|    Metric name                                      |    Description                                                           |
+-----------------------------------------------------+--------------------------------------------------------------------------+
| haproxy_backend_status                              | Current status of the service.                                           |
| haproxy_backend_current_sessions                    | Current number of active sessions.                                       |
| haproxy_backend_max_sessions                        | Maximum observed number of active sessions.                              |
| haproxy_backend_limit_sessions                      | Configured session limit.                                                |
| haproxy_backend_sessions_total                      | Total number of sessions.                                                |
| haproxy_backend_max_session_rate                    | Maximum observed number of sessions per second.                          |
| haproxy_backend_last_session_seconds                | Number of seconds since last session assigned to server/backend.         |
| haproxy_backend_current_queue                       | Current number of queued requests.                                       |
| haproxy_backend_max_queue                           | Maximum observed number of queued requests.                              |
| haproxy_backend_connection_attempts_total           | Total number of connection establishment attempts.                       |
| haproxy_backend_connection_reuses_total             | Total number of connection reuses.                                       |
| haproxy_backend_bytes_in_total                      | Current total of incoming bytes.                                         |
| haproxy_backend_bytes_out_total                     | Current total of outgoing bytes.                                         |
| haproxy_backend_queue_time_average_seconds          | Avg. queue time for last 1024 successful connections.                    |
| haproxy_backend_connect_time_average_seconds        | Avg. connect time for last 1024 successful connections.                  |
| haproxy_backend_response_time_average_seconds       | Avg. response time for last 1024 successful connections. (0 for TCP)     |
| haproxy_backend_total_time_average_seconds          | Avg. total time for last 1024 successful connections.                    |
| haproxy_backend_max_queue_time_seconds              | Maximum observed queue time.                                             |
| haproxy_backend_max_connect_time_seconds            | Maximum observed connect time.                                           |
| haproxy_backend_max_response_time_seconds           | Maximum observed response time. (0 for TCP)                              |
| haproxy_backend_max_total_time_seconds              | Maximum observed total time.                                             |
| haproxy_backend_requests_denied_total               | Total number of denied requests.                                         |
| haproxy_backend_responses_denied_total              | Total number of denied responses.                                        |
| haproxy_backend_connection_errors_total             | Total number of connection errors.                                       |
| haproxy_backend_response_errors_total               | Total number of response errors.                                         |
| haproxy_backend_retry_warnings_total                | Total number of retry warnings.                                          |
| haproxy_backend_redispatch_warnings_total           | Total number of redispatch warnings.                                     |
| haproxy_backend_failed_header_rewriting_total       | Total number of failed header rewriting warnings.                        |
| haproxy_backend_client_aborts_total                 | Total number of data transfers aborted by the client.                    |
| haproxy_backend_server_aborts_total                 | Total number of data transfers aborted by the server.                    |
| haproxy_backend_weight                              | Service weight.                                                          |
| haproxy_backend_uweight                             | Sum of active servers'user weights for a backend                         |
| haproxy_backend_active_servers                      | Current number of active servers.                                        |
| haproxy_backend_backup_servers                      | Current number of backup servers.                                        |
| haproxy_backend_check_up_down_total                 | Total number of UP->DOWN transitions.                                    |
| haproxy_backend_check_last_change_seconds           | Number of seconds since the last UP<->DOWN transition.                   |
| haproxy_backend_downtime_seconds_total              | Total downtime (in seconds) for the service.                             |
| haproxy_backend_loadbalanced_total                  | Total number of times a service was selected.                            |
| haproxy_backend_http_requests_total                 | Total number of HTTP requests received.                                  |
| haproxy_backend_http_responses_total                | Total number of HTTP responses.                                          |
| haproxy_backend_http_cache_lookups_total            | Total number of HTTP cache lookups.                                      |
| haproxy_backend_http_cache_hits_total               | Total number of HTTP cache hits.                                         |
| haproxy_backend_http_comp_bytes_in_total            | Total number of HTTP response bytes fed to the compressor.               |
| haproxy_backend_http_comp_bytes_out_total           | Total number of HTTP response bytes emitted by the compressor.           |
| haproxy_backend_http_comp_bytes_bypassed_total      | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit).  |
| haproxy_backend_http_comp_responses_total           | Total number of HTTP responses that were compressed.                     |
| haproxy_backend_internal_errors_total               | Total number of internal errors.                                         |
+-----------------------------------------------------+--------------------------------------------------------------------------+

* Server metrics

+----------------------------------------------------+---------------------------------------------------------------------------+
|    Metric name                                     |    Description                                                            |
+----------------------------------------------------+---------------------------------------------------------------------------+
| haproxy_server_status                              | Current status of the service.                                            |
| haproxy_server_current_sessions                    | Current number of active sessions.                                        |
| haproxy_server_max_sessions                        | Maximum observed number of active sessions.                               |
| haproxy_server_limit_sessions                      | Configured session limit.                                                 |
| haproxy_server_sessions_total                      | Total number of sessions.                                                 |
| haproxy_server_max_session_rate                    | Maximum observed number of sessions per second.                           |
| haproxy_server_last_session_seconds                | Number of seconds since last session assigned to server/backend.          |
| haproxy_server_current_queue                       | Current number of queued requests.                                        |
| haproxy_server_max_queue                           | Maximum observed number of queued requests.                               |
| haproxy_server_queue_limit                         | Configured maxqueue for the server (0 meaning no limit).                  |
| haproxy_server_bytes_in_total                      | Current total of incoming bytes.                                          |
| haproxy_server_bytes_out_total                     | Current total of outgoing bytes.                                          |
| haproxy_server_queue_time_average_seconds          | Avg. queue time for last 1024 successful connections.                     |
| haproxy_server_connect_time_average_seconds        | Avg. connect time for last 1024 successful connections.                   |
| haproxy_server_response_time_average_seconds       | Avg. response time for last 1024 successful connections. (0 for TCP)      |
| haproxy_server_total_time_average_seconds          | Avg. total time for last 1024 successful connections.                     |
| haproxy_server_max_queue_time_seconds              | Maximum observed queue time.                                              |
| haproxy_server_max_connect_time_seconds            | Maximum observed connect time.                                            |
| haproxy_server_max_response_time_seconds           | Maximum observed response time.  (0 for TCP)                              |
| haproxy_server_max_total_time_seconds              | Maximum observed total time.                                              |
| haproxy_server_connection_attempts_total           | Total number of connection establishment attempts.                        |
| haproxy_server_connection_reuses_total             | Total number of connection reuses.                                        |
| haproxy_server_responses_denied_total              | Total number of denied responses.                                         |
| haproxy_server_connection_errors_total             | Total number of connection errors.                                        |
| haproxy_server_response_errors_total               | Total number of response errors.                                          |
| haproxy_server_retry_warnings_total                | Total number of retry warnings.                                           |
| haproxy_server_redispatch_warnings_total           | Total number of redispatch warnings.                                      |
| haproxy_server_failed_header_rewriting_total       | Total number of failed header rewriting warnings.                         |
| haproxy_server_client_aborts_total                 | Total number of data transfers aborted by the client.                     |
| haproxy_server_server_aborts_total                 | Total number of data transfers aborted by the server.                     |
| haproxy_server_weight                              | Service weight.                                                           |
| haproxy_server_uweight                             | Server's user weight                                                      |
| haproxy_server_check_status                        | Status of last health check, if enabled. (see below for the mapping)      |
| haproxy_server_check_code                          | layer5-7 code, if available of the last health check.                     |
| haproxy_server_check_duration_seconds              | Total duration of the latest server health check, in seconds.             |
| haproxy_server_check_failures_total                | Total number of failed check (Only when the server is up).                |
| haproxy_server_check_up_down_total                 | Total number of UP->DOWN transitions.                                     |
| haproxy_server_downtime_seconds_total              | Total downtime (in seconds) for the service.                              |
| haproxy_server_check_last_change_seconds           | Number of seconds since the last UP<->DOWN transition.                    |
| haproxy_server_current_throttle                    | Current throttle percentage for the server, when slowstart is active.     |
| haproxy_server_loadbalanced_total                  | Total number of times a service was selected.                             |
| haproxy_server_http_responses_total                | Total number of HTTP responses.                                           |
| haproxy_server_idle_connections_current            | Current number of idle connections available for reuse.                   |
| haproxy_server_idle_connections_limit              | Limit on the number of available idle connections.                        |
| haproxy_server_internal_errors_total               | Total number of internal errors.                                          |
| haproxy_server_unsafe_idle_connections_current     | Current number of unsafe idle connections.                                |
| haproxy_server_safe_idle_connections_current       | Current number of safe idle connections.                                  |
| haproxy_server_used_connections_current            | Current number of connections in use.                                     |
| haproxy_server_need_connections_current            | Estimated needed number of connections.                                   |
+----------------------------------------------------+---------------------------------------------------------------------------+