2022-05-27 - Stream layers in HAProxy 2.6


1. Background

There are streams at plenty of levels in haproxy, essentially due to the
introduction of multiplexed protocols which provide high-level streams on top
of low-level streams, themselves either based on stream-oriented protocols or
datagram-oriented protocols.

The refactoring of the appctx and muxes that allowed to drop a lot of duplicate
code between 2.5 and 2.6-dev6 raised another concern with some entities like
"conn_stream" that were not specific to connections anymore, "endpoints" that
became entities on their own, and "targets" whose life had been extended to
last all along a connection.

It was time to rename all such legacy entities introduced in 1.8 and which had
turned particularly confusing over time as their roles evolved.


2. Naming principles

The global renaming of some entities between streams and connections was
articulated around several principles:

  - avoid the confusing use of "context" in shared places. For example, the
    endpoint's connection is in "ctx" and nothing makes it obvious that the
    endpoint's context is a connection, especially when an applet is there.

  - reserve relative nouns for pointers and not for types. "endpoint", just
    like "owner" or "peer" is relative, but when accessed from a different
    layer it starts to make no sense at all, or to make one believe it's
    something else, particularly with void*.

  - avoid too generic terms that have multiple meanings, or words that are
    synonyms in a same place (e.g. "peer" and "remote", or "endpoint" and
    "target"). If two synonyms are needed to designate two distinct entities,
    there's probably a problem elsewhere, or the problem is poorly defined.

  - make it clearer that all that is manipulated is related to streams. This
    particularly important in sample fetch functions for example, which tend
    to require low-level access and could be mislead in trying to follow the
    wrong chain when trying to get information about a connection.

  - use easily spellable short names that abbreviate unambiguously when used
    together in adjacent contexts


3. Current state as of 2.6

- when a name is required to designate the lower block that starts at the mux
  stream or the appctx, it is spoken of as a "stream endpoint", and abbreviated
  "se". It's okay because while "endpoint" itself is relative, "stream
  endpoint" unequivocally designates one extremity of a stream. If a type is
  needed for this in the future (e.g. via obj_type), then the type "stendp"
  may be used. Before 2.6-dev6 there was no name for this, it was known as
  conn_stream->ctx.

- the 2.6-dev6 cs_endpoint which preserves the state of a mux stream or an
  appctx and abstracts them in front of a conn_stream becomes a "stream
  endpoint descriptor", of type "sedesc" and often abbreviated "sd", "sed"
  or "ed". Its "target" pointer became "se" as per the rule above. Before
  2.6-dev6, these elements were mixed with others inside conn_stream. From
  the appctx it's called "sedesc" (few occurrences hence long name OK).

- the conn_stream which is always attached to either a stream or a health check
  and that is used to reach a mux or an applet becomes a "stream connector" of
  type "stconn", generally abbreviated "sc". Its "endp" pointer becomes
  "sedesc" as per the rule above, and that one has a back pointer "sc". The
  stream uses "scf" and "scb" as the respective front and back pointers to the
  stconns. Prior to 2.6-dev6, these parts were split between conn_stream and
  stream_interface.

- the sedesc's "ctx" which is solely used to store the connection as of now, is
  renamed "conn" to void any doubt in the context of applets or even muxes. In
  the future the connection should be attached to the "se" instead and this
  pointer should disappear (or be recycled for anything else).

The new 2.6 model looks like this:

                  +------------------------+
                  | stream or health check |
                  +------------------------+
                            ^   \ scf, scb
                           /     \
                          |       |
                           \     /
                        app \   v
                         +----------+
                         |  stconn  |
                         +----------+
                            ^   \ sedesc
                           /     \
                  . . . . | . . . | . . . . . split point (retries etc)
                           \     /
                         sc \   v
                         +----------+
                flags <--|  sedesc  |                      :  sedesc  :
                         +----------+              ...     +----------+
                   conn /   ^   \ se                           ^ \
     +------------+    /   /     \                             |  \
     | connection |<--'   |       |            ... OR ...      |   |
     +------------+        \     /                              \  |
      mux|  ^ |ctx       sd \   v                       : sedesc \ v
         |  | |   +----------------------+  \           #  +----------+ svcctx
         |  | |   | mux stream or appctx |   |          #  |  appctx  |--.
         |  | |   +----------------------+   |          #  +----------+  |
         |  | |           ^  |              /  private  #  :          :  |
         v  | |           |  v              >  to the   #  +----------+  |
   mux_ops  | |     +----------------+      \   mux     #  |  svcctx  |<-'
            | +---->| mux connection |       )          #  +----------+
            +------ +----------------+      /           #

Stream descriptors may exist in the following modes:
  - .conn = NULL, .se = NULL     : backend, not connection attempt yet
  - .conn = NULL, .se = <appctx> : frontend or backend, applet
  - .conn = <conn>, .se = NULL   : backend, connection in progress
  - .conn = <conn>, .se = <muxs> : frontend or backend, connected

Notes:
  - for historical reasons (connect, forced protocol upgrades, etc), during a
    connection setup or a rule-based protocol upgrade, the connection's "ctx"
    may temporarily point to the stconn


4. Invariants and cardinalities

Usually a stream is created from an existing stconn from a mux or some applets,
but may also be allocated first by other applets schedulers. After stream_new()
a stream always has exactly one stconn per side (scf, scb), each of which has
one ->sedesc. Each side is initialized with either one or no stream endpoint
attached to the descriptor.

Both applets and a mux stream always have a stream endpoint descriptor. AS SUCH
IT IS NEVER NECESSARY TO TEST FOR THE EXISTENCE OF THE SEDESC FROM ANY SIDE, IT
ALWAYS EXISTS. This explains why as much as possible it's preferable to use the
sedesc to access flags and statuses from any side, rather than bouncing via the
stconn.

An applet's app layer is always a stream, which means that there are always
channels accessible above, and there is always an opposite stream connector and
a stream endpoint descriptor. As such, it's always safe for an applet to access
the other side using sc_opposite().

When an outgoing connection is in the process of being established, the backend
side sedesc has its ->conn pointer pointing to the pending connection, and no
->se. Once the connection is established and a mux is chosen, it's attached to
the ->se. If an applet is used instead of a mux, the appctx is attached to the
sedesc's ->se and ->conn remains NULL.

If either side wants to detach from the other, it must allocate a new virgin
sedesc to replace the existing one, and leave the existing one to the endpoint,
since it continues to describe the stream endpoint. The stconn keeps its state
(modulo the updates related to the disconnection). The previous sedesc points
to a NULL stconn. For example, disconnecting from a backend mux will leave the
entities like this:

                                              +------------------------+
                                              | stream or health check |
                                              +------------------------+
                                                        ^   \ scf, scb
                                                       /     \
                                                      |       |
                                                       \     /
                                                    app \   v
                                                     +----------+
                                                     |  stconn  |
                                                     +----------+
                                                        ^   \ sedesc
                                                       /     \
                             NULL                     |       |
                              ^                        \     /
                           sc |               /      sc \   v
                         +----------+        /       +----------+
                flags <--|  sedesc1 |   . . . . .    |  sedesc2 |--> flags
                         +----------+      /         +----------+
                   conn /   ^   \ se      /       conn /     \ se
     +------------+    /   /     \                    |       |
     | connection |<--'   |       |                   v       v
     +------------+        \     /                  NULL     NULL
      mux|  ^ |ctx       sd \   v
         |  | |   +----------------------+
         |  | |   | mux stream or appctx |
         |  | |   +----------------------+
         |  | |           ^  |
         v  | |           |  v
   mux_ops  | |     +----------------+
            | +---->| mux connection |
            +------ +----------------+