BUG/MEDIUM: listener: Fix race condition when updating the global mngmt task
It is pretty similar to fbb934da90 ("BUG/MEDIUM: stick-table: fix a race
condition when updating the expiration task"). When the global management
task is running, at the end of its process function, it resets the expire
date by setting it to TICK_ETERNITY. In same time, a listener may reach a
global limit and decides to schedule the task. Thus it is possible to queue
the task and trigger the BUG_ON() on the expire date because its value was
set to TICK_ETERNITY in the means time:
FATAL: bug condition "task->expire == 0" matched at src/task.c:310
call trace(12):
| 0x662de8 [b8 01 00 00 00 c6 00 00]: __task_queue+0xc7/0x11e
| 0x63b03f [48 b8 04 00 00 00 05 00]: main+0x2535e
| 0x63ed1a [e9 d2 fd ff ff 48 8b 45]: listener_accept+0xf72/0xfda
| 0x6a36d3 [eb 01 90 c9 c3 55 48 89]: sock_accept_iocb+0x82/0x87
| 0x6af22f [48 8b 05 ca f9 13 00 8b]: fd_update_events+0x35a/0x497
| 0x42a7a8 [89 45 d8 83 7d d8 02 75]: main-0x1eb539
| 0x6158fb [48 8b 05 e6 06 1c 00 64]: run_poll_loop+0x2e7/0x319
| 0x615b6c [48 8b 05 ed 65 1d 00 48]: main-0x175
| 0x7ffff775bded [e9 69 fe ff ff 48 8b 4c]: libc:+0x8cded
| 0x7ffff77e1370 [48 89 c7 b8 3c 00 00 00]: libc:+0x112370
To fix the bug, a RW lock is introduced. It is used to fix the race
condition. A read lock is taken when the task is scheduled, in
listener_accpet() and a write lock is used at the end of process function to
set the expire date to TICK_ETERNITY. This lock should not be used very
often and most of time by "readers". So, the impact should be really
limited.
This patch should fix the issue #1930. It must be backported as far as 1.8
with some cautions because the code has evolved a lot since then.
(cherry picked from commit 13e86d947d1536dbe6d8f4057388727528546ae6)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 9cfdbf2dd7038673e49e13abe78265fa9c7d2b91)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 56630764d13a54321d89fc585a0970efe31a8eed)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
diff --git a/src/listener.c b/src/listener.c
index 6334974..007c686 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -45,6 +45,7 @@
/* list of the temporarily limited listeners because of lack of resource */
static struct mt_list global_listener_queue = MT_LIST_HEAD_INIT(global_listener_queue);
static struct task *global_listener_queue_task;
+static HA_RWLOCK_T global_listener_rwlock;
/* listener status for stats */
const char* li_status_st[LI_STATE_COUNT] = {
@@ -1131,7 +1132,9 @@
* later than <expire> ahead. The listener turns to LI_LIMITED.
*/
limit_listener(l, &global_listener_queue);
+ HA_RWLOCK_RDLOCK(LISTENER_LOCK, &global_listener_rwlock);
task_schedule(global_listener_queue_task, expire);
+ HA_RWLOCK_RDUNLOCK(LISTENER_LOCK, &global_listener_rwlock);
goto end;
limit_proxy:
@@ -1181,6 +1184,7 @@
/* very simple initialization, users will queue the task if needed */
global_listener_queue_task->context = NULL; /* not even a context! */
global_listener_queue_task->process = manage_global_listener_queue;
+ HA_RWLOCK_INIT(&global_listener_rwlock);
return 0;
}
@@ -1218,7 +1222,9 @@
dequeue_all_listeners();
out:
+ HA_RWLOCK_WRLOCK(LISTENER_LOCK, &global_listener_rwlock);
t->expire = TICK_ETERNITY;
+ HA_RWLOCK_WRUNLOCK(LISTENER_LOCK, &global_listener_rwlock);
task_queue(t);
return t;
}