BUG/MEDIUM: queue: deal with a rare TOCTOU in assign_server_and_queue()
After checking that a server or backend is full, it remains possible
to call pendconn_add() just after the last pending requests finishes, so
that there's no more connection on the server for very low maxconn (typ 1),
leaving new ones in queue till the timeout.
The approach depends on where the request was queued, though:
- when queued on a server, we can simply detect that we may dequeue
pending requests and wake them up, it will wake our request and
that's fine. This needs to be done in srv_redispatch_connect() when
the server is set.
- when queued on a backend, it means that all servers are done with
their requests. It means that all servers were full before the
check and all were empty after. In practice this will only concern
configs with less servers than threads. It's where the issue was
first spotted, and it's very hard to reproduce with more than one
server. In this case we need to load-balance again in order to find
a spare server (or even to fail). For this, we call the newly added
dedicated function pendconn_must_try_again() that tells whether or
not a blocked pending request was dequeued and needs to be retried.
This should be backported along with pendconn_must_try_again() to all
stable versions, but with extreme care because over time the queue's
locking evolved.
(cherry picked from commit 5541d4995d6d9e8e7956423d26c26bebe8f0eaea)
Signed-off-by: Willy Tarreau <w@1wt.eu>
(cherry picked from commit 68492650d35c8ec3d6352079b8d2b54fd43be1d6)
Signed-off-by: Willy Tarreau <w@1wt.eu>
(cherry picked from commit ad00496b1aeb68d90d737bec212eb9ca186e3c22)
Signed-off-by: Willy Tarreau <w@1wt.eu>
diff --git a/src/backend.c b/src/backend.c
index e12636f..65cadd7 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -964,6 +964,7 @@
struct server *srv;
int err;
+ balance_again:
if (s->pend_pos)
return SRV_STATUS_INTERNAL;
@@ -1032,8 +1033,18 @@
return SRV_STATUS_FULL;
p = pendconn_add(s);
- if (p)
+ if (p) {
+ /* There's a TOCTOU here: it may happen that between the
+ * moment we decided to queue the request and the moment
+ * it was done, the last active request on the server
+ * ended and no new one will be able to dequeue that one.
+ * Since we already have our server we don't care, this
+ * will be handled by the caller which will check for
+ * this condition and will immediately dequeue it if
+ * possible.
+ */
return SRV_STATUS_QUEUED;
+ }
else
return SRV_STATUS_INTERNAL;
}
@@ -1045,8 +1056,23 @@
case SRV_STATUS_FULL:
/* queue this stream into the proxy's queue */
p = pendconn_add(s);
- if (p)
+ if (p) {
+ /* There's a TOCTOU here: it may happen that between the
+ * moment we decided to queue the request and the moment
+ * it was done, the last active request in the backend
+ * ended and no new one will be able to dequeue that one.
+ * This is more visible with maxconn 1 where it can
+ * happen 1/1000 times, though the vast majority are
+ * correctly recovered from. Since it's so rare and we
+ * have no server assigned, the best solution in this
+ * case is to detect the condition, dequeue our request
+ * and balance it again.
+ */
+ if (unlikely(pendconn_must_try_again(p)))
+ goto balance_again;
+
return SRV_STATUS_QUEUED;
+ }
else
return SRV_STATUS_INTERNAL;
@@ -1922,7 +1948,16 @@
case SRV_STATUS_QUEUED:
s->conn_exp = tick_add_ifset(now_ms, s->be->timeout.queue);
s->scb->state = SC_ST_QUE;
- /* do nothing else and do not wake any other stream up */
+
+ /* handle the unlikely event where we added to the server's
+ * queue just after checking the server was full and before
+ * it released its last entry (with extremely low maxconn).
+ * Not needed for backend queues, already handled in
+ * assign_server_and_queue().
+ */
+ if (unlikely(srv && may_dequeue_tasks(srv, s->be)))
+ process_srv_queue(srv);
+
return 1;
case SRV_STATUS_INTERNAL: