Blame - contrib/prometheus-exporter/README - haproxy

2019-02-07 15:38:42 +0100

[diff] [blame]

1

PROMEX: A Prometheus exporter for HAProxy

2

-------------------------------------------

3

4

Prometheus is a monitoring and alerting system. More and more people use it to

5

monitor their environment (this is written February 2019). It collects metrics

6

from monitored targets by scraping metrics HTTP endpoints on these targets. For

Ilya Shipitsin

2020-03-14 17:47:28 +0500

[diff] [blame]

7

HAProxy, The Prometheus team officially supports an exporter written in Go

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

8

(https://github.com/prometheus/haproxy_exporter). But it requires an extra

9

software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus

10

exporter for HAProxy. It was developed as a service and is directly available in

11

HAProxy, like the stats applet.

12

13

However, PROMEX is not built by default with HAProxy. It is provided as an extra

Ilya Shipitsin

2020-03-14 17:47:28 +0500

[diff] [blame]

14

component for everyone want to use it. So you need to explicitly build HAProxy

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

15

with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance:

16

Willy Tarreau

d254aa8

2019-06-14 18:40:48 +0200

[diff] [blame]

17

> make TARGET=linux-glibc EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

18

19

if HAProxy provides the PROMEX service, the following build option will be

20

reported by the command "haproxy -vv":

21

22

Built with the Prometheus exporter as a service

23

24

To be used, it must be enabled in the configuration with an "http-request" rule

25

and the corresponding HTTP proxy must enable the HTX support. For instance:

frontend test

mode http

...

option http-use-htx

http-request use-service prometheus-exporter if { path /metrics }

...

This service has been developed as a third-party component because it could

36

become obsolete, depending on how much time Prometheus will remain heavily

37

used. This is said with no ulterior motive of course. Prometheus is a great

38

software and I hope all the well for it. But we involve in a environment moving

39

quickly and a solution may be obvious today could be deprecated the next

40

year. And because PROMEX is not integrated by default into the HAProxy codebase,

41

it will need some interest to be actively supported. All contribution of any

42

kind are welcome.

43

44

You must also be careful if you use with huge configurations. Unlike the stats

45

applet, all metrics are not grouped by service (proxy, listener or server). With

46

PROMEX, all lines for a given metric are provided as one single group. So

47

instead of collecting all metrics for a proxy before moving to the next one, we

48

must loop on all proxies for each metric. Same for the servers. Thus, it will

Ilya Shipitsin

2020-03-14 17:47:28 +0500

[diff] [blame]

49

spend much more resources to produce the Prometheus metrics than the CSV export

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

50

through the stats page. To give a comparison order, quick benchmarks shown that

51

a PROMEX dump is 5x slower and 20x more verbose than a CSV export.

52

Christopher Faulet

32d634f

2019-11-18 14:47:08 +0100

[diff] [blame]

metrics filtering

-------------------

It is possible to dynamically select the metrics to export if you don't use all

58

of them passing parameters in the query-string.

59

60

* Filtering on scopes

61

62

The metrics may be filtered by scopes. Multiple parameters with "scope" as name

63

may be passed in the query-string to filter exported metrics, with one of those

64

values: global, frontend, backend, server or '*' (means all). A scope parameter

65

with no value means to filter out all scopes (nothing is returned). The scope

66

parameters are parsed in their appearance order in the query-string. So an empty

67

scope will reset all scopes already parsed. But it can be overridden by

68

following scope parameters in the query-string. By default everything is

69

exported. Here are examples:

70

71

/metrics?scope=server # ==> server metrics will be exported

72

/metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported

73

/metrics?scope=*&scope= # ==> no metrics will be exported

74

/metrics?scope=&scope=global # ==> global metrics will be exported

75

Christopher Faulet

e48e996

2019-11-19 14:18:24 +0100

[diff] [blame]

76

* Filtering on servers state

77

78

It is possible to exclude from returned metrics all servers in maintenance mode

79

passing the parameter "no-maint" in the query-string. This parameter may help to

80

solve performance issues of configuration that use the server templates to

81

manage dynamic provisionning. Note there is no consistency check on the servers

82

state. So, if the state of a server changes while the exporter is running, only

83

a part of the metrics for this server will be dumped.

Christopher Faulet

32d634f

2019-11-18 14:47:08 +0100

[diff] [blame]

84

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

Exported metrics

------------------

* Globals metrics

+------------------------------------------------+-------------------------------------------------------------------------------+

91

| Metric name | Description |

92

+------------------------------------------------+-------------------------------------------------------------------------------+

93

| haproxy_process_nbthread | Configured number of threads. |

94

| haproxy_process_nbproc | Configured number of processes. |

95

| haproxy_process_relative_process_id | Relative process id, starting at 1. |

Christopher Faulet

8c8e4b1

2019-04-18 10:15:15 +0200

[diff] [blame]

96

| haproxy_process_start_time_seconds | Start time in seconds. |

97

| haproxy_process_max_memory_bytes | Per-process memory limit (in bytes); 0=unset. |

98

| haproxy_process_pool_allocated_bytes | Total amount of memory allocated in pools (in bytes). |

99

| haproxy_process_pool_used_bytes | Total amount of memory used in pools (in bytes). |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

100

| haproxy_process_pool_failures_total | Total number of failed pool allocations. |

101

| haproxy_process_max_fds | Maximum number of open file descriptors; 0=unset. |

Ilya Shipitsin

2020-03-14 17:47:28 +0500

[diff] [blame]

102

| haproxy_process_max_sockets | Maximum number of open sockets. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

103

| haproxy_process_max_connections | Maximum number of concurrent connections. |

104

| haproxy_process_hard_max_connections | Initial Maximum number of concurrent connections. |

105

| haproxy_process_current_connections | Number of active sessions. |

Christopher Faulet

8c8e4b1

2019-04-18 10:15:15 +0200

[diff] [blame]

106

| haproxy_process_connections_total | Total number of created sessions. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

107

| haproxy_process_requests_total | Total number of requests (TCP or HTTP). |

108

| haproxy_process_max_ssl_connections | Configured maximum number of concurrent SSL connections. |

109

| haproxy_process_current_ssl_connections | Number of opened SSL connections. |

110

| haproxy_process_ssl_connections_total | Total number of opened SSL connections. |

111

| haproxy_process_max_pipes | Configured maximum number of pipes. |

112

| haproxy_process_pipes_used_total | Number of pipes in used. |

113

| haproxy_process_pipes_free_total | Number of pipes unused. |

114

| haproxy_process_current_connection_rate | Current number of connections per second over last elapsed second. |

115

| haproxy_process_limit_connection_rate | Configured maximum number of connections per second. |

116

| haproxy_process_max_connection_rate | Maximum observed number of connections per second. |

117

| haproxy_process_current_session_rate | Current number of sessions per second over last elapsed second. |

118

| haproxy_process_limit_session_rate | Configured maximum number of sessions per second. |

119

| haproxy_process_max_session_rate | Maximum observed number of sessions per second. |

120

| haproxy_process_current_ssl_rate | Current number of SSL sessions per second over last elapsed second. |

121

| haproxy_process_limit_ssl_rate | Configured maximum number of SSL sessions per second. |

122

| haproxy_process_max_ssl_rate | Maximum observed number of SSL sessions per second. |

123

| haproxy_process_current_frontend_ssl_key_rate | Current frontend SSL Key computation per second over last elapsed second. |

124

| haproxy_process_max_frontend_ssl_key_rate | Maximum observed frontend SSL Key computation per second. |

125

| haproxy_process_frontent_ssl_reuse | SSL session reuse ratio (percent). |

126

| haproxy_process_current_backend_ssl_key_rate | Current backend SSL Key computation per second over last elapsed second. |

127

| haproxy_process_max_backend_ssl_key_rate | Maximum observed backend SSL Key computation per second. |

Christopher Faulet

8c8e4b1

2019-04-18 10:15:15 +0200

[diff] [blame]

128

| haproxy_process_ssl_cache_lookups_total | Total number of SSL session cache lookups. |

129

| haproxy_process_ssl_cache_misses_total | Total number of SSL session cache misses. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

130

| haproxy_process_http_comp_bytes_in_total | Number of bytes per second over last elapsed second, before http compression. |

131

| haproxy_process_http_comp_bytes_out_total | Number of bytes per second over last elapsed second, after http compression. |

132

| haproxy_process_limit_http_comp | Configured maximum input compression rate in bytes. |

133

| haproxy_process_current_zlib_memory | Current memory used for zlib in bytes. |

134

| haproxy_process_max_zlib_memory | Configured maximum amount of memory for zlib in bytes. |

135

| haproxy_process_current_tasks | Current number of tasks. |

136

| haproxy_process_current_run_queue | Current number of tasks in the run-queue. |

137

| haproxy_process_idle_time_percent | Idle to total ratio over last sample (percent). |

138

| haproxy_process_stopping | Non zero means stopping in progress. |

139

| haproxy_process_jobs | Current number of active jobs (listeners, sessions, open devices). |

140

| haproxy_process_unstoppable_jobs | Current number of active jobs that can't be stopped during a soft stop. |

141

| haproxy_process_listeners | Current number of active listeners. |

142

| haproxy_process_active_peers | Current number of active peers. |

143

| haproxy_process_connected_peers | Current number of connected peers. |

144

| haproxy_process_dropped_logs_total | Total number of dropped logs. |

145

| haproxy_process_busy_polling_enabled | Non zero if the busy polling is enabled. |

146

+------------------------------------------------+-------------------------------------------------------------------------------+

* Frontend metrics

+-------------------------------------------------+------------------------------------------------------------------------------+

151

| Metric name | Description |

152

+-------------------------------------------------+------------------------------------------------------------------------------+

153

| haproxy_frontend_status | Current status of the service. |

154

| haproxy_frontend_current_sessions | Current number of active sessions. |

155

| haproxy_frontend_max_sessions | Maximum observed number of active sessions. |

156

| haproxy_frontend_limit_sessions | Configured session limit. |

157

| haproxy_frontend_sessions_total | Total number of sessions. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

158

| haproxy_frontend_limit_session_rate | Configured limit on new sessions per second. |

159

| haproxy_frontend_max_session_rate | Maximum observed number of sessions per second. |

160

| haproxy_frontend_connections_rate_current | Current number of connections per second over the last elapsed second. |

161

| haproxy_frontend_connections_rate_max | Maximum observed number of connections per second. |

162

| haproxy_frontend_connections_total | Total number of connections. |

163

| haproxy_frontend_bytes_in_total | Current total of incoming bytes. |

164

| haproxy_frontend_bytes_out_total | Current total of outgoing bytes. |

165

| haproxy_frontend_requests_denied_total | Total number of denied requests. |

166

| haproxy_frontend_responses_denied_total | Total number of denied responses. |

167

| haproxy_frontend_request_errors_total | Total number of request errors. |

168

| haproxy_frontend_denied_connections_total | Total number of requests denied by "tcp-request connection" rules. |

169

| haproxy_frontend_denied_sessions_total | Total number of requests denied by "tcp-request session" rules. |

170

| haproxy_frontend_failed_header_rewriting_total | Total number of failed header rewriting warnings. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

171

| haproxy_frontend_http_requests_rate_max | Maximum observed number of HTTP requests per second. |

172

| haproxy_frontend_http_requests_total | Total number of HTTP requests received. |

173

| haproxy_frontend_http_responses_total | Total number of HTTP responses. |

174

| haproxy_frontend_intercepted_requests_total | Total number of intercepted HTTP requests. |

175

| haproxy_frontend_http_cache_lookups_total | Total number of HTTP cache lookups. |

176

| haproxy_frontend_http_cache_hits_total | Total number of HTTP cache hits. |

177

| haproxy_frontend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. |

178

| haproxy_frontend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. |

179

| haproxy_frontend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). |

180

| haproxy_frontend_http_comp_responses_total | Total number of HTTP responses that were compressed. |

181

+-------------------------------------------------+------------------------------------------------------------------------------+

182

Rick Rackow

a045ade

2019-10-08 11:28:06 +0200

[diff] [blame]

183

* Backend metrics

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

184

185

+-----------------------------------------------------+--------------------------------------------------------------------------+

186

| Metric name | Description |

187

+-----------------------------------------------------+--------------------------------------------------------------------------+

188

| haproxy_backend_status | Current status of the service. |

189

| haproxy_backend_current_sessions | Current number of active sessions. |

190

| haproxy_backend_max_sessions | Maximum observed number of active sessions. |

191

| haproxy_backend_limit_sessions | Configured session limit. |

192

| haproxy_backend_sessions_total | Total number of sessions. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

193

| haproxy_backend_max_session_rate | Maximum observed number of sessions per second. |

194

| haproxy_backend_last_session_seconds | Number of seconds since last session assigned to server/backend. |

195

| haproxy_backend_current_queue | Current number of queued requests. |

196

| haproxy_backend_max_queue | Maximum observed number of queued requests. |

197

| haproxy_backend_connection_attempts_total | Total number of connection establishment attempts. |

198

| haproxy_backend_connection_reuses_total | Total number of connection reuses. |

199

| haproxy_backend_bytes_in_total | Current total of incoming bytes. |

200

| haproxy_backend_bytes_out_total | Current total of outgoing bytes. |

Christopher Faulet

2019-11-08 15:12:29 +0100

[diff] [blame]

201

| haproxy_backend_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. |

202

| haproxy_backend_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. |

203

| haproxy_backend_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) |

204

| haproxy_backend_total_time_average_seconds | Avg. total time for last 1024 successful connections. |

Christopher Faulet

2019-11-08 15:05:31 +0100

[diff] [blame]

205

| haproxy_backend_max_queue_time_seconds | Maximum observed queue time. |

206

| haproxy_backend_max_connect_time_seconds | Maximum observed connect time. |

Christopher Faulet

2019-11-08 15:12:29 +0100

[diff] [blame]

207

| haproxy_backend_max_response_time_seconds | Maximum observed response time. (0 for TCP) |

Christopher Faulet

2019-11-08 15:05:31 +0100

[diff] [blame]

208

| haproxy_backend_max_total_time_seconds | Maximum observed total time. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

209

| haproxy_backend_requests_denied_total | Total number of denied requests. |

210

| haproxy_backend_responses_denied_total | Total number of denied responses. |

211

| haproxy_backend_connection_errors_total | Total number of connection errors. |

212

| haproxy_backend_response_errors_total | Total number of response errors. |

213

| haproxy_backend_retry_warnings_total | Total number of retry warnings. |

214

| haproxy_backend_redispatch_warnings_total | Total number of redispatch warnings. |

215

| haproxy_backend_failed_header_rewriting_total | Total number of failed header rewriting warnings. |

216

| haproxy_backend_client_aborts_total | Total number of data transfers aborted by the client. |

217

| haproxy_backend_server_aborts_total | Total number of data transfers aborted by the server. |

218

| haproxy_backend_weight | Service weight. |

219

| haproxy_backend_active_servers | Current number of active servers. |

220

| haproxy_backend_backup_servers | Current number of backup servers. |

221

| haproxy_backend_check_up_down_total | Total number of UP->DOWN transitions. |

222

| haproxy_backend_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. |

223

| haproxy_backend_downtime_seconds_total | Total downtime (in seconds) for the service. |

224

| haproxy_backend_loadbalanced_total | Total number of times a service was selected. |

225

| haproxy_backend_http_requests_total | Total number of HTTP requests received. |

226

| haproxy_backend_http_responses_total | Total number of HTTP responses. |

227

| haproxy_backend_http_cache_lookups_total | Total number of HTTP cache lookups. |

228

| haproxy_backend_http_cache_hits_total | Total number of HTTP cache hits. |

229

| haproxy_backend_http_comp_bytes_in_total | Total number of HTTP response bytes fed to the compressor. |

230

| haproxy_backend_http_comp_bytes_out_total | Total number of HTTP response bytes emitted by the compressor. |

231

| haproxy_backend_http_comp_bytes_bypassed_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). |

232

| haproxy_backend_http_comp_responses_total | Total number of HTTP responses that were compressed. |

233

+-----------------------------------------------------+--------------------------------------------------------------------------+

* Server metrics

+----------------------------------------------------+---------------------------------------------------------------------------+

238

| Metric name | Description |

239

+----------------------------------------------------+---------------------------------------------------------------------------+

240

| haproxy_server_status | Current status of the service. |

241

| haproxy_server_current_sessions | Current number of active sessions. |

242

| haproxy_server_max_sessions | Maximum observed number of active sessions. |

243

| haproxy_server_limit_sessions | Configured session limit. |

244

| haproxy_server_sessions_total | Total number of sessions. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

245

| haproxy_server_max_session_rate | Maximum observed number of sessions per second. |

246

| haproxy_server_last_session_seconds | Number of seconds since last session assigned to server/backend. |

247

| haproxy_server_current_queue | Current number of queued requests. |

248

| haproxy_server_max_queue | Maximum observed number of queued requests. |

249

| haproxy_server_queue_limit | Configured maxqueue for the server (0 meaning no limit). |

250

| haproxy_server_bytes_in_total | Current total of incoming bytes. |

251

| haproxy_server_bytes_out_total | Current total of outgoing bytes. |

Christopher Faulet

2019-11-08 15:12:29 +0100

[diff] [blame]

252

| haproxy_server_queue_time_average_seconds | Avg. queue time for last 1024 successful connections. |

253

| haproxy_server_connect_time_average_seconds | Avg. connect time for last 1024 successful connections. |

254

| haproxy_server_response_time_average_seconds | Avg. response time for last 1024 successful connections. (0 for TCP) |

255

| haproxy_server_total_time_average_seconds | Avg. total time for last 1024 successful connections. |

Christopher Faulet

2019-11-08 15:05:31 +0100

[diff] [blame]

256

| haproxy_server_max_queue_time_seconds | Maximum observed queue time. |

257

| haproxy_server_max_connect_time_seconds | Maximum observed connect time. |

Christopher Faulet

2019-11-08 15:12:29 +0100

[diff] [blame]

258

| haproxy_server_max_response_time_seconds | Maximum observed response time. (0 for TCP) |

Christopher Faulet

2019-11-08 15:05:31 +0100

[diff] [blame]

259

| haproxy_server_max_total_time_seconds | Maximum observed total time. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

260

| haproxy_server_connection_attempts_total | Total number of connection establishment attempts. |

261

| haproxy_server_connection_reuses_total | Total number of connection reuses. |

262

| haproxy_server_responses_denied_total | Total number of denied responses. |

263

| haproxy_server_connection_errors_total | Total number of connection errors. |

264

| haproxy_server_response_errors_total | Total number of response errors. |

265

| haproxy_server_retry_warnings_total | Total number of retry warnings. |

266

| haproxy_server_redispatch_warnings_total | Total number of redispatch warnings. |

267

| haproxy_server_failed_header_rewriting_total | Total number of failed header rewriting warnings. |

268

| haproxy_server_client_aborts_total | Total number of data transfers aborted by the client. |

269

| haproxy_server_server_aborts_total | Total number of data transfers aborted by the server. |

270

| haproxy_server_weight | Service weight. |

Christopher Faulet

e019d18

2019-11-21 14:35:46 +0100

[diff] [blame]

271

| haproxy_server_check_status | Status of last health check, if enabled. (see below for the mapping) |

272

| haproxy_server_check_code | layer5-7 code, if available of the last health check. |

Christopher Faulet

4ab0efb

2020-02-27 16:12:07 +0100

[diff] [blame]

273

| haproxy_server_check_duration_seconds | Total duration of the latest server health check, in seconds. |

Christopher Faulet

2019-02-07 15:38:42 +0100

[diff] [blame]

274

| haproxy_server_check_failures_total | Total number of failed check (Only when the server is up). |

275

| haproxy_server_check_up_down_total | Total number of UP->DOWN transitions. |

276

| haproxy_server_downtime_seconds_total | Total downtime (in seconds) for the service. |

277

| haproxy_server_check_last_change_seconds | Number of seconds since the last UP<->DOWN transition. |

278

| haproxy_server_current_throttle | Current throttle percentage for the server, when slowstart is active. |

279

| haproxy_server_loadbalanced_total | Total number of times a service was selected. |

280

| haproxy_server_http_responses_total | Total number of HTTP responses. |

Christopher Faulet

bd767f7

2019-11-08 15:24:32 +0100

[diff] [blame]

281

| haproxy_server_idle_connections_current | Current number of idle connections available for reuse. |

282

| haproxy_server_idle_connections_limit | Limit on the number of available idle connections. |

Christopher Faulet