BUG/MINOR: quic: fix race on quic_conns list during affinity rebind
Each quic_conn are attached in a global thread-local quic_conns list
used for "show quic" command. During thread rebinding, a connection is
detached from its local list instance and moved to its new thread list.
However this operation is not thread-safe and may cause a race
condition.
To fix this, only remove the connection from its list inside
qc_set_tid_affinity(). The connection is inserted only after in
qc_finalize_affinity_rebind() on the new thread instance thus prevented
a race condition. One impact of this is that a connection will be
invisible during rebinding for "show quic".
A connection must not transition to closing state in between this two
steps or else cleanup via quic_handle_stopping() may not miss it. To
ensure this, this patch relies on the previous commit :
commit d6646dddccb1aae08f60717b5b6743c513c37299
MINOR: quic: finalize affinity change as soon as possible
This should be backported up to 2.7.
diff --git a/src/quic_conn.c b/src/quic_conn.c
index 254ce58..33ac151 100644
--- a/src/quic_conn.c
+++ b/src/quic_conn.c
@@ -8524,12 +8524,10 @@
fd_migrate_on(qc->fd, new_tid);
}
- /* Remove conn from per-thread list instance. */
+ /* Remove conn from per-thread list instance. It will be hidden from
+ * "show quic" until rebinding is completed.
+ */
qc_detach_th_ctx_list(qc, 0);
- /* Connection must not be closing or else it must be inserted in quic_conns_clo list instance instead. */
- BUG_ON(qc->flags & (QUIC_FL_CONN_CLOSING|QUIC_FL_CONN_DRAINING));
- LIST_APPEND(&ha_thread_ctx[new_tid].quic_conns, &qc->el_th_ctx);
- qc->qc_epoch = HA_ATOMIC_LOAD(&qc_epoch);
node = eb64_first(&qc->cids);
BUG_ON(!node || eb64_next(node)); /* One and only one CID must be present before affinity rebind. */
@@ -8569,6 +8567,16 @@
BUG_ON(!(qc->flags & QUIC_FL_CONN_AFFINITY_CHANGED));
qc->flags &= ~QUIC_FL_CONN_AFFINITY_CHANGED;
+ /* A connection must not pass to closing state until affinity rebind
+ * is completed. Else quic_handle_stopping() may miss it during process
+ * stopping cleanup.
+ */
+ BUG_ON(qc->flags & (QUIC_FL_CONN_CLOSING|QUIC_FL_CONN_DRAINING));
+
+ /* Reinsert connection in ha_thread_ctx global list. */
+ LIST_APPEND(&th_ctx->quic_conns, &qc->el_th_ctx);
+ qc->qc_epoch = HA_ATOMIC_LOAD(&qc_epoch);
+
/* Reactivate FD polling if connection socket is active. */
qc_want_recv(qc);