MEDIUM: map: make the "clear map" operation yield
As reported in issue #419, a "clear map" operation on a very large map
can take a lot of time and freeze the entire process for several seconds.
This patch makes sure that pat_ref_prune() can regularly yield after
clearing some entries so that the rest of the process continues to work.
The first part, the removal of the patterns, can take quite some time
by itself in one run but it's still relatively fast. It may block for
up to 100ms for 16M IP addresses in a tree typically. This change needed
to declare an I/O handler for the clear operation so that we can get
back to it after yielding.
The second part can be much slower because it deconstructs the elements
and its users, but it iterates progressively so we can yield less often
here.
The patch was tested with traffic in parallel sollicitating the map being
released and showed no problem. Some traffic will definitely notice an
incomplete map but the filling is already not atomic anyway thus this is
not different.
It may be backported to stable versions once sufficiently tested for side
effects, at least as far as 2.0 in order to avoid the watchdog triggering
when the process is frozen there. For a better behaviour, all these
prune_* functions should support yielding so that the callers have a
chance to continue also yield in turn.
(cherry picked from commit d1d005d7f6b894ed243c177937626d888b3d03e1)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 5baab7460cf976c26a0afa8433ff41a36b5d9a4e)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
diff --git a/src/pattern.c b/src/pattern.c
index a82900b..3dbc285 100644
--- a/src/pattern.c
+++ b/src/pattern.c
@@ -2100,18 +2100,27 @@
}
/* This function prune all entries of <ref>. This function
- * prune the associated pattern_expr.
+ * prunes the associated pattern_expr. It may return before the end of
+ * the list is reached, returning 0, to yield. The caller must call it
+ * again. Otherwise it returns 1 once done.
*/
-void pat_ref_prune(struct pat_ref *ref)
+int pat_ref_prune(struct pat_ref *ref)
{
struct pat_ref_elt *elt, *safe;
struct pattern_expr *expr;
struct bref *bref, *back;
+ int loops = 0;
list_for_each_entry(expr, &ref->pat, list) {
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
expr->pat_head->prune(expr);
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
+ loops++;
+ /* yield often, some lists may be huge, especially those
+ * having to be freed through free_pattern_tree()
+ */
+ if (loops > 10)
+ return 0;
}
/* we trash pat_ref_elt in a second time to ensure that data is
@@ -2132,9 +2141,11 @@
free(elt->pattern);
free(elt->sample);
free(elt);
+ loops++;
+ if (loops > 100000)
+ return 0;
}
-
-
+ return 1;
}
/* This function lookup for existing reference <ref> in pattern_head <head>. */