MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.
This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.
With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:
stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
http-request track-sc0 base # track host+path, ignore query string
http-request return status 503 content-type text/html \
lf-file excuse.html if { sc0_http_fail_rate gt 10 }
A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.
Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
diff --git a/doc/configuration.txt b/doc/configuration.txt
index c281459..12b87ad 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -10838,6 +10838,19 @@
http_err_cnt above for what is accounted as an error). The result is an
integer which can be matched using ACLs.
+ - http_fail_cnt : HTTP Failure Count. It is a positive 32-bit integer which
+ counts the absolute number of HTTP response failures induced by servers
+ which matched this entry. Errors are counted on invalid and truncated
+ responses, as well as any 5xx response other than 501 or 505. It aims at
+ being used combined with path or URI to detect service failures.
+
+ - http_fail_rate(<period>) : frequency counter (takes 12 bytes). It takes
+ an integer parameter <period> which indicates in milliseconds the length
+ of the period over which the average is measured. It reports the average
+ HTTP response failure rate over that period, in requests per period (see
+ http_fail_cnt above for what is accounted as a failure). The result is an
+ integer which can be matched using ACLs.
+
- bytes_in_cnt : client to server byte count. It is a positive 64-bit
integer which counts the cumulative number of bytes received from clients
which matched this entry. Headers are included in the count. This may be
@@ -16162,6 +16175,21 @@
period configured in the table. See also the sc_http_err_rate sample fetch
keyword.
+table_http_fail_cnt(<table>)
+ Uses the string representation of the input sample to perform a look up in
+ the specified table. If the key is not found in the table, integer value zero
+ is returned. Otherwise the converter returns the cumulative number of HTTP
+ failures associated with the input sample in the designated table. See also
+ the sc_http_fail_cnt sample fetch keyword.
+
+table_http_fail_rate(<table>)
+ Uses the string representation of the input sample to perform a look up in
+ the specified table. If the key is not found in the table, integer value zero
+ is returned. Otherwise the average rate of HTTP failures associated with the
+ input sample in the designated table, measured in amount of failures over the
+ period configured in the table. See also the sc_http_fail_rate sample fetch
+ keyword.
+
table_http_req_cnt(<table>)
Uses the string representation of the input sample to perform a look up in
the specified table. If the key is not found in the table, integer value zero
@@ -17166,6 +17194,23 @@
includes the both request errors and 4xx error responses. See also
src_http_err_rate.
+sc_http_fail_cnt(<ctr>[,<table>]) : integer
+sc0_http_fail_cnt([<table>]) : integer
+sc1_http_fail_cnt([<table>]) : integer
+sc2_http_fail_cnt([<table>]) : integer
+ Returns the cumulative number of HTTP response failures from the currently
+ tracked counters. This includes the both response errors and 5xx status codes
+ other than 501 and 505. See also src_http_fail_cnt.
+
+sc_http_fail_rate(<ctr>[,<table>]) : integer
+sc0_http_fail_rate([<table>]) : integer
+sc1_http_fail_rate([<table>]) : integer
+sc2_http_fail_rate([<table>]) : integer
+ Returns the average rate of HTTP response failures from the currently tracked
+ counters, measured in amount of failures over the period configured in the
+ table. This includes the both response errors and 5xx status codes other than
+ 501 and 505. See also src_http_fail_rate.
+
sc_http_req_cnt(<ctr>[,<table>]) : integer
sc0_http_req_cnt([<table>]) : integer
sc1_http_req_cnt([<table>]) : integer
@@ -17402,6 +17447,21 @@
includes the both request errors and 4xx error responses. If the address is
not found, zero is returned. See also sc/sc0/sc1/sc2_http_err_rate.
+src_http_fail_cnt([<table>]) : integer
+ Returns the cumulative number of HTTP response failures triggered by the
+ incoming connection's source address in the current proxy's stick-table or in
+ the designated stick-table. This includes the both repsonse errors and 5xx
+ status codes other than 501 and 505. See also sc/sc0/sc1/sc2_http_fail_cnt.
+ If the address is not found, zero is returned.
+
+src_http_fail_rate([<table>]) : integer
+ Returns the average rate of HTTP response failures triggered by the incoming
+ connection's source address in the current proxy's stick-table or in the
+ designated stick-table, measured in amount of failures over the period
+ configured in the table. This includes the both response errors and 5xx
+ status codes other than 501 and 505. If the address is not found, zero is
+ returned. See also sc/sc0/sc1/sc2_http_fail_rate.
+
src_http_req_cnt([<table>]) : integer
Returns the cumulative number of HTTP requests from the incoming connection's
source address in the current proxy's stick-table or in the designated stick-