DOC: commit a few old design thoughts files
These ones were design notes and ideas collected during the 1.5
development phase lying on my development machine. There might still
be some value in keeping them for future reference since they mention
certain corner cases.
diff --git a/doc/design-thoughts/binding-possibilities.txt b/doc/design-thoughts/binding-possibilities.txt
new file mode 100644
index 0000000..3f5e432
--- /dev/null
+++ b/doc/design-thoughts/binding-possibilities.txt
@@ -0,0 +1,167 @@
+2013/10/10 - possibilities for setting source and destination addresses
+
+
+When establishing a connection to a remote device, this device is designated
+as a target, which designates an entity defined in the configuration. A same
+target appears only once in a configuration, and multiple targets may share
+the same settings if needed.
+
+The following types of targets are currently supported :
+
+ - listener : all connections with this type of target come from clients ;
+ - server : connections to such targets are for "server" lines ;
+ - peer : connections to such target address "peer" lines in "peers"
+ sections ;
+ - proxy : these targets are used by "dispatch", "option transparent"
+ or "option http_proxy" statements.
+
+A connection might not be reused between two different targets, even if all
+parameters seem similar. One of the reason is that some parameters are specific
+to the target and are not easy or not cheap to compare (eg: bind to interface,
+mss, ...).
+
+A number of source and destination addresses may be set for a given target.
+
+ - listener :
+ - the "from" address:port is set by accept()
+
+ - the "to" address:port is set if conn_get_to_addr() is called
+
+ - peer :
+ - the "from" address:port is not set
+
+ - the "to" address:port is static and dependent only on the peer
+
+ - server :
+ - the "from" address may be set alone when "source" is used with
+ a forced IP address, or when "usesrc clientip" is used.
+
+ - the "from" port may be set only combined with the address when
+ "source" is used with IP:port, IP:port-range or "usesrc client" is
+ used. Note that in this case, both the address and the port may be
+ 0, meaning that the kernel will pick the address or port and that
+ the final value might not match the one explicitly set (eg:
+ important for logging).
+
+ - the "from" address may be forced from a header which implies it
+ may change between two consecutive requests on the same connection.
+
+ - the "to" address and port are set together when connecting to a
+ regular server, or by copying the client's IP address when
+ "server 0.0.0.0" is used. Note that the destination port may be
+ an offset applied to the original destination port.
+
+ - proxy :
+ - the "from" address may be set alone when "source" is used with a
+ forced IP address or when "usesrc clientip" is used.
+
+ - the "from" port may be set only combined with the address when
+ "source" is used with IP:port or with "usesrc client". There is
+ no ip:port range for a proxy as of now. Same comment applies as
+ above when port and/or address are 0.
+
+ - the "from" address may be forced from a header which implies it
+ may change between two consecutive requests on the same connection.
+
+ - the "to" address and port are set together, either by configuration
+ when "dispatch" is used, or dynamically when "transparent" is used
+ (1:1 with client connection) or "option http_proxy" is used, where
+ each client request may lead to a different destination address.
+
+
+At the moment, there are some limits in what might happen between multiple
+concurrent requests to a same target.
+
+ - peers parameter do not change, so no problem.
+
+ - server parameters may change in this way :
+ - a connection may require a source bound to an IP address found in a
+ header, which will fall back to the "source" settings if the address
+ is not found in this header. This means that the source address may
+ switch between a dynamically forced IP address and another forced
+ IP and/or port range.
+
+ - if the element is not found (eg: header), the remaining "forced"
+ source address might very well be empty (unset), so the connection
+ reuse is acceptable when switching in that direction.
+
+ - it is not possible to switch between client and clientip or any of
+ these and hdr_ip() because they're exclusive.
+
+ - using a source address/port belonging to a port range is compatible
+ with connection reuse because there is a single range per target, so
+ switching from a range to another range means we remain in the same
+ range.
+
+ - destination address may currently not change since the only possible
+ case for dynamic destination address setting is the transparent mode,
+ reproducing the client's destination address.
+
+ - proxy parameters may change in this way :
+ - a connection may require a source bound to an IP address found in a
+ header, which will fall back to the "source" settings if the address
+ is not found in this header. This means that the source address may
+ switch between a dynamically forced IP address and another forced
+ IP and/or port range.
+
+ - if the element is not found (eg: header), the remaining "forced"
+ source address might very well be empty (unset), so the connection
+ reuse is acceptable when switching in that direction.
+
+ - it is not possible to switch between client and clientip or any of
+ these and hdr_ip() because they're exclusive.
+
+ - proxies do not support port ranges at the moment.
+
+ - destination address might change in the case where "option http_proxy"
+ is used.
+
+So, for each source element (IP, port), we want to know :
+ - if the element was assigned by static configuration (eg: ":80")
+ - if the element was assigned from a connection-specific value (eg: usesrc clientip)
+ - if the element was assigned from a configuration-specific range (eg: 1024-65535)
+ - if the element was assigned from a request-specific value (eg: hdr_ip(xff))
+ - if the element was not assigned at all
+
+For the destination, we want to know :
+ - if the element was assigned by static configuration (eg: ":80")
+ - if the element was assigned from a connection-specific value (eg: transparent)
+ - if the element was assigned from a request-specific value (eg: http_proxy)
+
+We don't need to store the information about the origin of the dynamic value
+since we have the value itself. So in practice we have :
+ - default value, unknown (not yet checked with getsockname/getpeername)
+ - default value, known (check done)
+ - forced value (known)
+ - forced range (known)
+
+We can't do that on an ip:port basis because the port may be fixed regardless
+of the address and conversely.
+
+So that means :
+
+ enum {
+ CO_ADDR_NONE = 0, /* not set, unknown value */
+ CO_ADDR_KNOWN = 1, /* not set, known value */
+ CO_ADDR_FIXED = 2, /* fixed value, known */
+ CO_ADDR_RANGE = 3, /* from assigned range, known */
+ } conn_addr_values;
+
+ unsigned int new_l3_src_status:2;
+ unsigned int new_l4_src_status:2;
+ unsigned int new_l3_dst_status:2;
+ unsigned int new_l4_dst_status:2;
+
+ unsigned int cur_l3_src_status:2;
+ unsigned int cur_l4_src_status:2;
+ unsigned int cur_l3_dsp_status:2;
+ unsigned int cur_l4_dst_status:2;
+
+ unsigned int new_family:2;
+ unsigned int cur_family:2;
+
+Note: this obsoletes CO_FL_ADDR_FROM_SET and CO_FL_ADDR_TO_SET. These flags
+must be changed to individual l3+l4 checks ORed between old and new values,
+or better, set to cur only which will inherit new.
+
+In the connection, these values may be merged in the same word as err_code.
diff --git a/doc/design-thoughts/connection-reuse.txt b/doc/design-thoughts/connection-reuse.txt
new file mode 100644
index 0000000..0c90eb2
--- /dev/null
+++ b/doc/design-thoughts/connection-reuse.txt
@@ -0,0 +1,139 @@
+2013/10/17 - server connection management and reuse
+
+Current state
+-------------
+
+At the moment, a connection entity is needed to carry any address
+information. This means in the following situations, we need a server
+connection :
+
+- server is elected and the server's destination address is set
+
+- transparent mode is elected and the destination address is set from
+ the incoming connection
+
+- proxy mode is enabled, and the destination's address is set during
+ the parsing of the HTTP request
+
+- connection to the server fails and must be retried on the same
+ server using the same parameters, especially the destination
+ address (SN_ADDR_SET not removed)
+
+
+On the accepting side, we have further requirements :
+
+- allocate a clean connection without a stream interface
+
+- incrementally set the accepted connection's parameters without
+ clearing it, and keep track of what is set (eg: getsockname).
+
+- initialize a stream interface in established mode
+
+- attach the accepted connection to a stream interface
+
+
+This means several things :
+
+- the connection has to be allocated on the fly the first time it is
+ needed to store the source or destination address ;
+
+- the connection has to be attached to the stream interface at this
+ moment ;
+
+- it must be possible to incrementally set some settings on the
+ connection's addresses regardless of the connection's current state
+
+- the connection must not be released across connection retries ;
+
+- it must be possible to clear a connection's parameters for a
+ redispatch without having to detach/attach the connection ;
+
+- we need to allocate a connection without an existing stream interface
+
+So on the accept() side, it looks like this :
+
+ fd = accept();
+ conn = new_conn();
+ get_some_addr_info(&conn->addr);
+ ...
+ si = new_si();
+ si_attach_conn(si, conn);
+ si_set_state(si, SI_ST_EST);
+ ...
+ get_more_addr_info(&conn->addr);
+
+On the connect() side, it looks like this :
+
+ si = new_si();
+ while (!properly_connected) {
+ if (!(conn = si->end)) {
+ conn = new_conn();
+ conn_clear(conn);
+ si_attach_conn(si, conn);
+ }
+ else {
+ if (connected) {
+ f = conn->flags & CO_FL_XPRT_TRACKED;
+ conn->flags &= ~CO_FL_XPRT_TRACKED;
+ conn_close(conn);
+ conn->flags |= f;
+ }
+ if (!correct_dest)
+ conn_clear(conn);
+ }
+ set_some_addr_info(&conn->addr);
+ si_set_state(si, SI_ST_CON);
+ ...
+ set_more_addr_info(&conn->addr);
+ conn->connect();
+ if (must_retry) {
+ close_conn(conn);
+ }
+ }
+
+Note: we need to be able to set the control and transport protocols.
+On outgoing connections, this is set once we know the destination address.
+On incoming connections, this is set the earliest possible (once we know
+the source address).
+
+The problem analysed below was solved on 2013/10/22
+
+| ==> the real requirement is to know whether a connection is still valid or not
+| before deciding to close it. CO_FL_CONNECTED could be enough, though it
+| will not indicate connections that are still waiting for a connect to occur.
+| This combined with CO_FL_WAIT_L4_CONN and CO_FL_WAIT_L6_CONN should be OK.
+|
+| Alternatively, conn->xprt could be used for this, but needs some careful checks
+| (it's used by conn_full_close at least).
+|
+| Right now, conn_xprt_close() checks conn->xprt and sets it to NULL.
+| conn_full_close() also checks conn->xprt and sets it to NULL, except
+| that the check on ctrl is performed within xprt. So conn_xprt_close()
+| followed by conn_full_close() will not close the file descriptor.
+| Note that conn_xprt_close() is never called, maybe we should kill it ?
+|
+| Note: at the moment, it's problematic to leave conn->xprt to NULL before doing
+| xprt_init() because we might end up with a pending file descriptor. Or at
+| least with some transport not de-initialized. We might thus need
+| conn_xprt_close() when conn_xprt_init() fails.
+|
+| The fd should be conditionned by ->ctrl only, and the transport layer by ->xprt.
+|
+| - conn_prepare_ctrl(conn, ctrl)
+| - conn_prepare_xprt(conn, xprt)
+| - conn_prepare_data(conn, data)
+|
+| Note: conn_xprt_init() needs conn->xprt so it's not a problem to set it early.
+|
+| One problem might be with conn_xprt_close() not being able to know if xprt_init()
+| was called or not. That's where it might make sense to only set ->xprt during init.
+| Except that it does not fly with outgoing connections (xprt_init is called after
+| connect()).
+|
+| => currently conn_xprt_close() is only used by ssl_sock.c and decides whether
+| to do something based on ->xprt_ctx which is set by ->init() from xprt_init().
+| So there is nothing to worry about. We just need to restore conn_xprt_close()
+| and rely on ->ctrl to close the fd instead of ->xprt.
+|
+| => we have the same issue with conn_ctrl_close() : when is the fd supposed to be
+| valid ? On outgoing connections, the control is set much before the fd...
diff --git a/doc/design-thoughts/entities-v2.txt b/doc/design-thoughts/entities-v2.txt
new file mode 100644
index 0000000..8c9eb48
--- /dev/null
+++ b/doc/design-thoughts/entities-v2.txt
@@ -0,0 +1,276 @@
+2012/07/05 - Connection layering and sequencing
+
+
+An FD has a state :
+ - CLOSED
+ - READY
+ - ERROR (?)
+ - LISTEN (?)
+
+A connection has a state :
+ - CLOSED
+ - ACCEPTED
+ - CONNECTING
+ - ESTABLISHED
+ - ERROR
+
+A stream interface has a state :
+ - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
+
+Note that CON and CER might be replaced by EST if the connection state is used
+instead. CON might even be more suited than EST to indicate that a connection
+is known.
+
+
+si_shutw() must do :
+
+ data_shutw()
+ if (shutr) {
+ data_close()
+ ctrl_shutw()
+ ctrl_close()
+ }
+
+si_shutr() must do :
+ data_shutr()
+ if (shutw) {
+ data_close()
+ ctrl_shutr()
+ ctrl_close()
+ }
+
+Each of these steps may fail, in which case the step must be retained and the
+operations postponed in an asynchronous task.
+
+The first asynchronous data_shut() might already fail so it is mandatory to
+save the other side's status with the connection in order to let the async task
+know whether the 3 next steps must be performed.
+
+The connection (or perhaps the FD) needs to know :
+ - the desired close operations : DSHR, DSHW, CSHR, CSHW
+ - the completed close operations : DSHR, DSHW, CSHR, CSHW
+
+
+On the accept() side, we probably need to know :
+ - if a header is expected (eg: accept-proxy)
+ - if this header is still being waited for
+ => maybe both info might be combined into one bit
+
+ - if a data-layer accept() is expected
+ - if a data-layer accept() has been started
+ - if a data-layer accept() has been performed
+ => possibly 2 bits, to indicate the need to free()
+
+On the connect() side, we need to konw :
+ - the desire to send a header (eg: send-proxy)
+ - if this header has been sent
+ => maybe both info might be combined
+
+ - if a data-layer connect() is expected
+ - if a data-layer connect() has been started
+ - if a data-layer connect() has been completed
+ => possibly 2 bits, to indicate the need to free()
+
+On the response side, we also need to know :
+ - the desire to send a header (eg: health check response for monitor-net)
+ - if this header was sent
+ => might be the same as sending a header over a new connection
+
+Note: monitor-net has precedence over proxy proto and data layers. Same for
+ health mode.
+
+For multi-step operations, use 2 bits :
+ 00 = operation not desired, not performed
+ 10 = operation desired, not started
+ 11 = operation desired, started but not completed
+ 01 = operation desired, started and completed
+
+ => X != 00 ==> operation desired
+ X & 01 ==> operation at least started
+ X & 10 ==> operation not completed
+
+Note: no way to store status information for error reporting.
+
+Note2: it would be nice if "tcp-request connection" rules could work at the
+connection level, just after headers ! This means support for tracking stick
+tables, possibly not too much complicated.
+
+
+Proposal for incoming connection sequence :
+
+- accept()
+- if monitor-net matches or if mode health => try to send response
+- if accept-proxy, wait for proxy request
+- if tcp-request connection, process tcp rules and possibly keep the
+ pointer to stick-table
+- if SSL is enabled, switch to SSL handshake
+- then switch to DATA state and instantiate a session
+
+We just need a map of handshake handlers on the connection. They all manage the
+FD status themselves and set the callbacks themselves. If their work succeeds,
+they remove themselves from the list. If it fails, they remain subscribed and
+enable the required polling until they are woken up again or the timeout strikes.
+
+Identified handshake handlers for incoming connections :
+ - HH_HEALTH (tries to send OK and dies)
+ - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
+ - HH_SEND_OK (tries to send "OK" and dies)
+ - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
+ - HH_ACCEPT_PROXY (waits for PROXY line and parses it)
+ - HH_TCP_RULES (processes TCP rules)
+ - HH_SSL_HS (starts SSL handshake)
+ - HH_ACCEPT_SESSION (instanciates a session)
+
+Identified handshake handlers for outgoing connections :
+ - HH_SEND_PROXY (tries to build and send the PROXY line)
+ - HH_SSL_HS (starts SSL handshake)
+
+For the pollers, we could check that handshake handlers are not 0 and decide to
+call a generic connection handshake handler instead of usual callbacks. Problem
+is that pollers don't know connections, they know fds. So entities which manage
+handlers should update change the FD callbacks accordingly.
+
+With a bit of care, we could have :
+ - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
+ => merges HEALTH, SEND_OK and SEND_HTTP_OK
+
+It sounds like the ctrl vs data state for the connection are per-direction
+(eg: support an async ctrl shutw while still reading data).
+
+Also support shutr/shutw status at L4/L7.
+
+In practice, what we really need is :
+
+shutdown(conn) =
+ conn.data.shut()
+ conn.ctrl.shut()
+ conn.fd.shut()
+
+close(conn) =
+ conn.data.close()
+ conn.ctrl.close()
+ conn.fd.close()
+
+With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
+
+ HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
+
+The connection has to be closed at 3 places after a successful response :
+ - DATA (RSSL over RTCP)
+ - CTRL (RTCP to close connection to server)
+ - SOCK (FD to close connection to second process)
+
+Externally, the connection is seen with very few flags :
+ - SHR
+ - SHW
+ - ERR
+
+We don't need a CLOSED flag as a connection must always be detached when it's closed.
+
+The internal status doesn't need to be exposed :
+ - FD allocated (Y/N)
+ - CTRL initialized (Y/N)
+ - CTRL connected (Y/N)
+ - CTRL handlers done (Y/N)
+ - CTRL failed (Y/N)
+ - CTRL shutr (Y/N)
+ - CTRL shutw (Y/N)
+ - DATA initialized (Y/N)
+ - DATA connected (Y/N)
+ - DATA handlers done (Y/N)
+ - DATA failed (Y/N)
+ - DATA shutr (Y/N)
+ - DATA shutw (Y/N)
+
+(note that having flags for operations needing to be completed might be easier)
+--------------
+
+Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
+very unlikely since the only functions manipulating this are in the code of
+the data/ctrl handlers.
+
+FDSET/FDCLR cannot be directly controlled by the stream interface since it also
+depends on the DATA layer (WANT_READ/WANT_WRITE).
+
+But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).
+
+Example: an SSL conn relies on an FD. The buffer is full, and wants the conn to
+stop reading. It must not stop the FD itself. It is the read function which
+should notice that it has nothing to do with a read wake-up, which needs to
+disable reading.
+
+Conversely, when calling conn->chk_rcv(), the reader might get a WANT_READ or
+even WANT_WRITE and adjust the FDs accordingly.
+
+------------------------
+
+OK, the problem is simple : we don't manipulate the FD at the right level.
+We should have :
+ ->connect(), ->chk_snd(), ->chk_rcv(), ->shutw(), ->shutr() which are
+ called from the upper layer (buffer)
+ ->recv(), ->send(), called from the lower layer
+
+Note that the SHR is *reported* by lower layer but can be forced by upper
+layer. In this case it's like a delayed abort. The difficulty consists in
+knowing the output data were correctly read. Probably we'd need to drain
+incoming data past the active shutr().
+
+The only four purposes of the top-down shutr() call are :
+ - acknowledge a shut read report : could probably be done better
+ - read timeout => disable reading : it's a delayed abort. We want to
+ report that the buffer is SHR, maybe even the connection, but the
+ FD clearly isn't.
+ - read abort due to error on the other side or desire to close (eg:
+ http-server-close) : delayed abort
+ - complete abort
+
+The active shutr() is problematic as we can't disable reading if we expect some
+exchanges for data acknowledgement. We probably need to drain data only until
+the shutw() has been performed and ACKed.
+
+A connection shut down for read would behave like this :
+
+ 1) bidir exchanges
+
+ 2) shutr() => read_abort_pending=1
+
+ 3) drain input, still send output
+
+ 4) shutw()
+
+ 5) drain input, wait for read0 or ack(shutw)
+
+ 6) close()
+
+--------------------- 2012/07/05 -------------------
+
+Communications must be performed this way :
+
+ connection <-> channel <-> connection
+
+A channel is composed of flags and stats, and may store data in either a buffer
+or a pipe. We need low-layer operations between sockets and buffers or pipes.
+Right now we only support sockets, but later we might support remote sockets
+and maybe pipes or shared memory segments.
+
+So we need :
+
+ - raw_sock_to_buf() => receive raw data from socket into buffer
+ - raw_sock_to_pipe => receive raw data from socket into pipe (splice in)
+ - raw_sock_from_buf() => send raw data from buffer to socket
+ - raw_sock_from_pipe => send raw data from pipe to socket (splice out)
+
+ - ssl_sock_to_buf() => receive ssl data from socket into buffer
+ - ssl_sock_to_pipe => receive ssl data from socket into a pipe (NULL)
+ - ssl_sock_from_buf() => send ssl data from buffer to socket
+ - ssl_sock_from_pipe => send ssl data from pipe to socket (NULL)
+
+These functions should set such status flags :
+
+#define ERR_IN 0x01
+#define ERR_OUT 0x02
+#define SHUT_IN 0x04
+#define SHUT_OUT 0x08
+#define EMPTY_IN 0x10
+#define FULL_OUT 0x20
+