MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.
This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.
With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:
stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
http-request track-sc0 base # track host+path, ignore query string
http-request return status 503 content-type text/html \
lf-file excuse.html if { sc0_http_fail_rate gt 10 }
A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.
Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
diff --git a/src/http_act.c b/src/http_act.c
index dee738b..5a51ca4 100644
--- a/src/http_act.c
+++ b/src/http_act.c
@@ -1777,10 +1777,10 @@
struct stktable *t;
struct stksess *ts;
struct stktable_key *key;
- void *ptr1, *ptr2, *ptr3, *ptr4;
+ void *ptr1, *ptr2, *ptr3, *ptr4, *ptr5, *ptr6;
int opt;
- ptr1 = ptr2 = ptr3 = ptr4 = NULL;
+ ptr1 = ptr2 = ptr3 = ptr4 = ptr5 = ptr6 = NULL;
opt = ((rule->from == ACT_F_HTTP_REQ) ? SMP_OPT_DIR_REQ : SMP_OPT_DIR_RES) | SMP_OPT_FINAL;
t = rule->arg.trk_ctr.table.t;
@@ -1810,7 +1810,13 @@
ptr4 = stktable_data_ptr(t, ts, STKTABLE_DT_HTTP_ERR_RATE);
}
- if (ptr1 || ptr2 || ptr3 || ptr4) {
+ if (rule->from == ACT_F_HTTP_RES && (unsigned)(s->txn->status - 500) < 100 &&
+ s->txn->status != 501 && s->txn->status != 505) {
+ ptr5 = stktable_data_ptr(t, ts, STKTABLE_DT_HTTP_FAIL_CNT);
+ ptr6 = stktable_data_ptr(t, ts, STKTABLE_DT_HTTP_FAIL_RATE);
+ }
+
+ if (ptr1 || ptr2 || ptr3 || ptr4 || ptr5 || ptr6) {
HA_RWLOCK_WRLOCK(STK_SESS_LOCK, &ts->lock);
if (ptr1)
@@ -1823,6 +1829,11 @@
if (ptr4)
update_freq_ctr_period(&stktable_data_cast(ptr4, http_err_rate),
t->data_arg[STKTABLE_DT_HTTP_ERR_RATE].u, 1);
+ if (ptr5)
+ stktable_data_cast(ptr5, http_fail_cnt)++;
+ if (ptr6)
+ update_freq_ctr_period(&stktable_data_cast(ptr6, http_fail_rate),
+ t->data_arg[STKTABLE_DT_HTTP_FAIL_RATE].u, 1);
HA_RWLOCK_WRUNLOCK(STK_SESS_LOCK, &ts->lock);