BUG/MEDIUM: spoe: Kill applets if there are pending connections and nbthread > 1

When the processing stage is finished for a SPOE applet, before returning it
into the idle list, we check if the assigned server appears as full or if
there are some pending connections on the backend or the assigned server. If
yes, it means we reach a maxconn and we close the applet to free a
slot. Otherwise, the applet can be reused. This test is only performed if
there are more than one thread.

It is important to close SPOE applets when there are pending connections for
multithreaded instances because connections with the SPOE agents are
persistent and local to a thread (applets are local to a thread). If a
maxconn is configured, some threads may take all available slots for a
while, leaving remaining threads without any free slot to process SPOE
messages. It is especially true if the maxconn is low.

This patch should fix the issue #705. It must be backported as far as
1.8. However, the code in 1.8 is quite different, a test must be performed
to be sure it works well.

(cherry picked from commit 9e647e5af77faa6a95dd511d78ed8763781e764a)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 1b28e0a57fe2393370c5f64450ce2f27a4fabeb7)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 02bfbacb84ceafc28dc79f416da51b97943f01e0)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
(cherry picked from commit 93a494373eeaebc56e014db1743015f788ec2f79)
Signed-off-by: Christopher Faulet <cfaulet@haproxy.com>
diff --git a/src/flt_spoe.c b/src/flt_spoe.c
index a4b3c8c..213985d 100644
--- a/src/flt_spoe.c
+++ b/src/flt_spoe.c
@@ -1464,8 +1464,6 @@
 			goto stop;
 
 		default:
-			/* HELLO handshake is finished, set the idle timeout and
-			 * add the applet in the list of running applets. */
 			_HA_ATOMIC_ADD(&agent->counters.idles, 1);
 			appctx->st0 = SPOE_APPCTX_ST_IDLE;
 			SPOE_APPCTX(appctx)->node.key = 0;
@@ -1753,6 +1751,20 @@
 	}
 
 	if (appctx->st0 == SPOE_APPCTX_ST_PROCESSING && SPOE_APPCTX(appctx)->cur_fpa < agent->max_fpa) {
+		struct server *srv = objt_server(si_strm(si)->target);
+
+		/* With several threads, close the applet if there are pending
+		 * connections or if the server is full. Otherwise, add the
+		 * applet in the idle list.
+		 */
+		if (global.nbthread > 1 &&
+		    (agent->b.be->nbpend ||
+		     (srv && (srv->nbpend || (srv->maxconn && srv->served >=srv_dynamic_maxconn(srv)))))) {
+			SPOE_APPCTX(appctx)->status_code = SPOE_FRM_ERR_NONE;
+			appctx->st0 = SPOE_APPCTX_ST_DISCONNECT;
+			appctx->st1 = SPOE_APPCTX_ERR_NONE;
+			goto next;
+		}
 		_HA_ATOMIC_ADD(&agent->counters.idles, 1);
 		appctx->st0 = SPOE_APPCTX_ST_IDLE;
 		eb32_insert(&agent->rt[tid].idle_applets, &SPOE_APPCTX(appctx)->node);