BUG/MINOR: pools: always flush pools about to be destroyed

When destroying a pool (e.g. at exit or when resizing buffers), it's
important to try to free all their local objects otherwise we can leave
some in the cache. This is particularly visible when changing "bufsize",
because "show pools" will then show two "trash" pools, one of which
contains a single object in cache (which is fortunately not reachable).
In all cases this happens while single-threaded so that's easy to do,
we just have to do it on the current thread.

The easiest way to do this is to pass an extra argument to function
pool_evict_from_local_cache() to force a full flush instead of a
partial one.

This can probably be backported to about all branches where this
applies, but at least 2.4 needs it.

(cherry picked from commit c895c441c7579db652e4ed976c14c2f5b2de0c0e)
[wt: context adjustments as 2.5 doesn't use pool_evict_last_items()]
Signed-off-by: Willy Tarreau <w@1wt.eu>
(cherry picked from commit cc5504f86409b754317bdc2d4dc8e2c554a25549)
Signed-off-by: Willy Tarreau <w@1wt.eu>
diff --git a/include/haproxy/pool.h b/include/haproxy/pool.h
index 4d3bc7f..00ea859 100644
--- a/include/haproxy/pool.h
+++ b/include/haproxy/pool.h
@@ -73,7 +73,7 @@
 extern THREAD_LOCAL size_t pool_cache_bytes;   /* total cache size */
 extern THREAD_LOCAL size_t pool_cache_count;   /* #cache objects   */
 
-void pool_evict_from_local_cache(struct pool_head *pool);
+void pool_evict_from_local_cache(struct pool_head *pool, int full);
 void pool_evict_from_local_caches();
 void pool_put_to_cache(struct pool_head *pool, void *ptr);
 
diff --git a/src/pool.c b/src/pool.c
index f15a4a8..5de4547 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -267,13 +267,14 @@
  * we don't want a single cache to use all the cache for itself). For this, the
  * list is scanned in reverse.
  */
-void pool_evict_from_local_cache(struct pool_head *pool)
+void pool_evict_from_local_cache(struct pool_head *pool, int full)
 {
 	struct pool_cache_head *ph = &pool->cache[tid];
 	struct pool_cache_item *item;
 
-	while (ph->count >= 16 + pool_cache_count / 8 &&
-	       pool_cache_bytes > CONFIG_HAP_POOL_CACHE_SIZE * 3 / 4) {
+	while ((ph->count && full) ||
+	       (ph->count >= 16 + pool_cache_count / 8 &&
+		pool_cache_bytes > CONFIG_HAP_POOL_CACHE_SIZE * 3 / 4)) {
 		item = LIST_NEXT(&ph->list, typeof(item), by_pool);
 		ph->count--;
 		pool_cache_bytes -= pool->size;
@@ -330,7 +331,7 @@
 
 	if (unlikely(pool_cache_bytes > CONFIG_HAP_POOL_CACHE_SIZE * 3 / 4)) {
 		if (ph->count >= 16 + pool_cache_count / 8)
-			pool_evict_from_local_cache(pool);
+			pool_evict_from_local_cache(pool, 0);
 		if (pool_cache_bytes > CONFIG_HAP_POOL_CACHE_SIZE)
 			pool_evict_from_local_caches();
 	}
@@ -469,6 +470,9 @@
 void *pool_destroy(struct pool_head *pool)
 {
 	if (pool) {
+#ifdef CONFIG_HAP_POOLS
+		pool_evict_from_local_cache(pool, 1);
+#endif
 		pool_flush(pool);
 		if (pool->used)
 			return pool;