| ----------------------------------------------- |
| Stream Processing Offload Engine (SPOE) |
| Version 1.2 |
| ( Last update: 2017-09-22 ) |
| ----------------------------------------------- |
| Author : Christopher Faulet |
| Contact : cfaulet at haproxy dot com |
| |
| |
| SUMMARY |
| -------- |
| |
| 0. Terms |
| 1. Introduction |
| 2. SPOE configuration |
| 2.1. SPOE scope |
| 2.2. "spoe-agent" section |
| 2.3. "spoe-message" section |
| 2.4. "spoe-group" section |
| 2.5. Example |
| 3. SPOP specification |
| 3.1. Data types |
| 3.2. Frames |
| 3.2.1. Frame capabilities |
| 3.2.2. Frame types overview |
| 3.2.3. Workflow |
| 3.2.4. Frame: HAPROXY-HELLO |
| 3.2.5. Frame: AGENT-HELLO |
| 3.2.6. Frame: NOTIFY |
| 3.2.7. Frame: ACK |
| 3.2.8. Frame: HAPROXY-DISCONNECT |
| 3.2.9. Frame: AGENT-DISCONNECT |
| 3.3. Events & messages |
| 3.4. Actions |
| 3.5. Errors & timeouts |
| 4. Logging |
| |
| |
| 0. Terms |
| --------- |
| |
| * SPOE : Stream Processing Offload Engine. |
| |
| A SPOE is a filter talking to servers managed ba a SPOA to offload the |
| stream processing. An engine is attached to a proxy. A proxy can have |
| several engines. Each engine is linked to an agent and only one. |
| |
| * SPOA : Stream Processing Offload Agent. |
| |
| A SPOA is a service that will receive info from a SPOE to offload the |
| stream processing. An agent manages several servers. It uses a backend to |
| reference all of them. By extension, these servers can also be called |
| agents. |
| |
| * SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA |
| servers. |
| |
| This protocol is used by engines to talk to agents. It is an in-house |
| binary protocol described in this documentation. |
| |
| |
| 1. Introduction |
| ---------------- |
| |
| SPOE is a feature introduced in HAProxy 1.7. It makes possible the |
| communication with external components to retrieve some info. The idea started |
| with the problems caused by most ldap libs not working fine in event-driven |
| systems (often at least the connect() is blocking). So, it is hard to properly |
| implement Single Sign On solution (SSO) in HAProxy. The SPOE will ease this |
| kind of processing, or we hope so. |
| |
| Now, the aim of SPOE is to allow any kind of offloading on the streams. First |
| releases, besides being experimental, won't do lot of things. As we will see, |
| there are few handled events and even less actions supported. Actually, for |
| now, the SPOE can offload the processing before "tcp-request content", |
| "tcp-response content", "http-request" and "http-response" rules. And it only |
| supports variables definition. But, in spite of these limited features, we can |
| easily imagine to implement SSO solution, ip reputation or ip geolocation |
| services. |
| |
| |
| 2. SPOE configuration |
| ---------------------- |
| |
| Because SPOE is implemented as a filter, To use it, you must declare a "filter |
| spoe" line in a proxy section (frontend/backend/listen) : |
| |
| frontend my-front |
| ... |
| filter spoe [engine <name>] config <file> |
| ... |
| |
| The "config" parameter is mandatory. It specififies the SPOE configuration |
| file. The engine name is optional. It can be set to declare the scope to use in |
| the SPOE configuration. So it is possible to use the same SPOE configuration |
| for several engines. If no name is provided, the SPOE configuration must not |
| contain any scope directive. |
| |
| We use a separate configuration file on purpose. By commenting SPOE filter |
| line, you completely disable the feature, including the parsing of sections |
| reserved to SPOE. This is also a way to keep the HAProxy configuration clean. |
| |
| A SPOE configuration file must contains, at least, the SPOA configuration |
| ("spoe-agent" section) and SPOE messages/groups ("spoe-message" or "spoe-group" |
| sections) attached to this agent. |
| |
| IMPORTANT : The configuration of a SPOE filter must be located in a dedicated |
| file. But the backend used by a SPOA must be declared in HAProxy configuration |
| file. |
| |
| 2.1. SPOE scope |
| ------------------------- |
| |
| If you specify an engine name on the SPOE filter line, then you need to define |
| scope in the SPOE configuration with the same name. You can have several SPOE |
| scope in the same file. In each scope, you must define one and only one |
| "spoe-agent" section to configure the SPOA linked to your SPOE and several |
| "spoe-message" and "spoe-group" sections to describe, respecively, messages and |
| group of messages sent to servers mananged by your SPOA. |
| |
| A SPOE scope starts with this kind of line : |
| |
| [<name>] |
| |
| where <name> is the same engine name specified on the SPOE filter line. The |
| scope ends when the file ends or when another scope is found. |
| |
| Example : |
| [my-first-engine] |
| spoe-agent my-agent |
| ... |
| spoe-message msg1 |
| ... |
| spoe-message msg2 |
| ... |
| spoe-group grp1 |
| ... |
| spoe-group grp2 |
| ... |
| |
| [my-second-engine] |
| ... |
| |
| If no engine name is provided on the SPOE filter line, no SPOE scope must be |
| found in the SPOE configuration file. All the file is considered to be in the |
| same anonymous and implicit scope. |
| |
| The engine name must be uniq for a proxy. If no engine name is provided on the |
| SPOE filter line, the SPOE agent name is used by default. |
| |
| 2.2. "spoe-agent" section |
| -------------------------- |
| |
| For each engine, you must define one and only one "spoe-agent" section. In this |
| section, you will declare SPOE messages and the backend you will use. You will |
| also set timeouts and options to customize your agent's behaviour. |
| |
| |
| spoe-agent <name> |
| Create a new SPOA with the name <name>. It must have one and only one |
| "spoe-agent" definition by SPOE scope. |
| |
| Arguments : |
| <name> is the name of the agent section. |
| |
| following keywords are supported : |
| - groups |
| - log |
| - maxconnrate |
| - maxerrrate |
| - max-frame-size |
| - max-waiting-frames |
| - messages |
| - [no] option async |
| - [no] option dontlog-normal |
| - [no] option pipelining |
| - [no] option send-frag-payload |
| - option continue-on-error |
| - option force-set-var |
| - option set-on-error |
| - option set-process-time |
| - option set-total-time |
| - option var-prefix |
| - register-var-names |
| - timeout hello|idle|processing |
| - use-backend |
| |
| |
| groups <grp-name> ... |
| Declare the list of SPOE groups that an agent will handle. |
| |
| Arguments : |
| <grp-name> is the name of a SPOE group. |
| |
| Groups declared here must be found in the same engine scope, else an error is |
| triggered during the configuration parsing. You can have many "groups" lines. |
| |
| See also: "spoe-group" section. |
| |
| |
| log global |
| log <address> [len <length>] [format <format>] <facility> [<level> [<minlevel>]] |
| no log |
| Enable per-instance logging of events and traffic. |
| |
| Prefix : |
| no should be used when the logger list must be flushed. |
| |
| See the HAProxy Configuration Manual for details about this option. |
| |
| maxconnrate <number> |
| Set the maximum number of connections per second to <number>. The SPOE will |
| stop to open new connections if the maximum is reached and will wait to |
| acquire an existing one. So it is important to set "timeout hello" to a |
| relatively small value. |
| |
| |
| maxerrrate <number> |
| Set the maximum number of errors per second to <number>. The SPOE will stop |
| its processing if the maximum is reached. |
| |
| |
| max-frame-size <number> |
| Set the maximum allowed size for frames exchanged between HAProxy and SPOA. |
| It must be in the range [256, tune.bufsize-4] (4 bytes are reserved for the |
| frame length). By default, it is set to (tune.bufsize-4). |
| |
| max-waiting-frames <number> |
| Set the maximum number of frames waiting for an acknowledgement on the same |
| connection. This value is only used when the pipelinied or asynchronus |
| exchanges between HAProxy and SPOA are enabled. By default, it is set to 20. |
| |
| messages <msg-name> ... |
| Declare the list of SPOE messages that an agent will handle. |
| |
| Arguments : |
| <msg-name> is the name of a SPOE message. |
| |
| Messages declared here must be found in the same engine scope, else an error |
| is triggered during the configuration parsing. You can have many "messages" |
| lines. |
| |
| See also: "spoe-message" section. |
| |
| |
| option async |
| no option async |
| Enable or disable the support of asynchronus exchanges between HAProxy and |
| SPOA. By default, this option is enabled. |
| |
| |
| option continue-on-error |
| Do not stop the events processing when an error occurred on a stream. |
| |
| By default, for a specific stream, when an abnormal/unexpected error occurs, |
| the SPOE is disabled for all the transaction. So if you have several events |
| configured, such error on an event will disabled all followings. For TCP |
| streams, this will disable the SPOE for the whole session. For HTTP streams, |
| this will disable it for the transaction (request and response). |
| |
| When set, this option bypass this behaviour and only the current event will |
| be ignored. |
| |
| |
| option dontlog-normal |
| no option dontlog-normal |
| Enable or disable logging of normal, successful processing. |
| |
| Arguments : none |
| |
| See also: "log" and section 4 about logging. |
| |
| |
| option force-set-var |
| By default, SPOE filter only register already known variables (mainly from |
| parsing of the configuration). If you want that haproxy trusts the agent and |
| registers all variables (ex: can be useful for LUA workload), activate this |
| option. |
| |
| Caution : this option opens to a variety of attacks such as a rogue SPOA that |
| asks to register too many variables. |
| |
| |
| option pipelining |
| no option pipelining |
| Enable or disable the support of pipelined exchanges between HAProxy and |
| SPOA. By default, this option is enabled. |
| |
| |
| option send-frag-payload |
| no option send-frag-payload |
| Enable or disable the sending of fragmented payload to SPOA. By default, this |
| option is enabled. |
| |
| |
| option set-on-error <var name> |
| Define the variable to set when an error occurred during an event processing. |
| |
| Arguments : |
| |
| <var name> is the variable name, without the scope. The name may only |
| contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. |
| |
| This variable will only be set when an error occurred in the scope of the |
| transaction. As for all other variables define by the SPOE, it will be |
| prefixed. So, if your variable name is "error" and your prefix is |
| "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". |
| |
| When set, the variable is an integer representing the error reason. For values |
| under 256, it represents an error coming from the engine. Below 256, it |
| reports a SPOP error. In this case, to retrieve the right SPOP status code, |
| you must remove 256 to this value. Here are possible values: |
| |
| * 1 a timeout occurred during the event processing. |
| |
| * 2 an error was triggered during the resources allocation. |
| |
| * 3 the frame payload exceeds the frame size and it cannot be |
| fragmented. |
| |
| * 4 the fragmentation of a payload is aborted. |
| |
| * 5 The frame processing has been interrupted by HAProxy. |
| |
| * 255 an unknown error occurred during the event processing. |
| |
| * 256+N a SPOP error occurred during the event processing (see section |
| "Errors & timeouts"). |
| |
| Note that if "option continue-on-error" is set, the variable is not |
| automatically removed between events processing. |
| |
| See also: "option continue-on-error", "option var-prefix". |
| |
| |
| option set-process-time <var name> |
| Define the variable to set to report the processing time of the last event or |
| group. |
| |
| Arguments : |
| |
| <var name> is the variable name, without the scope. The name may only |
| contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. |
| |
| This variable will be set in the scope of the transaction. As for all other |
| variables define by the SPOE, it will be prefixed. So, if your variable name |
| is "process_time" and your prefix is "my_spoe_pfx", the variable will be |
| "txn.my_spoe_pfx.process_time". |
| |
| When set, the variable is an integer representing the delay to process the |
| event or the group, in milliseconds. From the stream point of view, it is the |
| latency added by the SPOE processing for the last handled event or group. |
| |
| If several events or groups are processed for the same stream, this value |
| will be overrideen. |
| |
| See also: "option set-total-time". |
| |
| |
| option set-total-time <var name> |
| Define the variable to set to report the total processing time SPOE for a |
| stream. |
| |
| Arguments : |
| |
| <var name> is the variable name, without the scope. The name may only |
| contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. |
| |
| This variable will be set in the scope of the transaction. As for all other |
| variables define by the SPOE, it will be prefixed. So, if your variable name |
| is "total_time" and your prefix is "my_spoe_pfx", the variable will be |
| "txn.my_spoe_pfx.total_time". |
| |
| When set, the variable is an integer representing the sum of processing times |
| for a stream, in milliseconds. From the stream point of view, it is the |
| latency added by the SPOE processing. |
| |
| If several events or groups are processed for the same stream, this value |
| will be updated. |
| |
| See also: "option set-process-time". |
| |
| |
| option var-prefix <prefix> |
| Define the prefix used when variables are set by an agent. |
| |
| Arguments : |
| |
| <prefix> is the prefix used to limit the scope of variables set by an |
| agent. |
| |
| To avoid conflict with other variables defined by HAProxy, all variables |
| names will be prefixed. By default, the "spoe-agent" name is used. This |
| option can be used to customize it. |
| |
| The prefix will be added between the variable scope and its name, separated |
| by a '.'. It may only contain characters 'a-z', 'A-Z', '0-9', '.' and '_', as |
| for variables name. In HAProxy configuration, you need to use this prefix as |
| a part of the variables name. For example, if an agent define the variable |
| "myvar" in the "txn" scope, with the prefix "my_spoe_pfx", then you should |
| use "txn.my_spoe_pfx.myvar" name in your HAProxy configuration. |
| |
| By default, an agent will never set new variables at runtime: It can only set |
| new value for existing ones. If you want a different behaviour, see |
| force-set-var option and register-var-names directive. |
| |
| register-var-names <var name> ... |
| Register some variable names. By default, an agent will not be allowed to set |
| new variables at runtime. This rule can be totally relaxed by setting the |
| option "force-set-var". If you know all the variables you will need, this |
| directive is a good way to register them without letting an agent doing what |
| it want. This is only required if these variables are not referenced anywhere |
| in the HAProxy configuration or the SPOE one. |
| |
| Arguments: |
| <var name> is a variable name without the scope. The name may only |
| contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. |
| |
| The prefix will be automatically added during the registration. You can have |
| many "register-var-names" lines. |
| |
| See also: "option force-set-var", "option var-prefix". |
| |
| timeout hello <timeout> |
| Set the maximum time to wait for an agent to receive the AGENT-HELLO frame. |
| It is applied on the stream that handle the connection with the agent. |
| |
| Arguments : |
| <timeout> is the timeout value specified in milliseconds by default, but |
| can be in any other unit if the number is suffixed by the unit, |
| as explained at the top of this document. |
| |
| This timeout is an applicative timeout. It differ from "timeout connect" |
| defined on backends. |
| |
| |
| timeout idle <timeout> |
| Set the maximum time to wait for an agent to close an idle connection. It is |
| applied on the stream that handle the connection with the agent. |
| |
| Arguments : |
| <timeout> is the timeout value specified in milliseconds by default, but |
| can be in any other unit if the number is suffixed by the unit, |
| as explained at the top of this document. |
| |
| |
| timeout processing <timeout> |
| Set the maximum time to wait for a stream to process an event, i.e to acquire |
| a stream to talk with an agent, to encode all messages, to send the NOTIFY |
| frame, to receive the corrsponding acknowledgement and to process all |
| actions. It is applied on the stream that handle the client and the server |
| sessions. |
| |
| Arguments : |
| <timeout> is the timeout value specified in milliseconds by default, but |
| can be in any other unit if the number is suffixed by the unit, |
| as explained at the top of this document. |
| |
| |
| use-backend <backend> |
| Specify the backend to use. It must be defined. |
| |
| Arguments : |
| <backend> is the name of a valid "backend" section. |
| |
| |
| 2.3. "spoe-message" section |
| ---------------------------- |
| |
| To offload the stream processing, SPOE will send messages with specific |
| information at a specific moment in the stream life and will wait for |
| corresponding replies to know what to do. |
| |
| |
| spoe-message <name> |
| Create a new SPOE message with the name <name>. |
| |
| Arguments : |
| <name> is the name of the SPOE message. |
| |
| Here you define a message that can be referenced in a "spoe-agent" |
| section. Following keywords are supported : |
| - acl |
| - args |
| - event |
| |
| See also: "spoe-agent" section. |
| |
| |
| acl <aclname> <criterion> [flags] [operator] <value> ... |
| Declare or complete an access list. |
| |
| See section 7 about ACL usage in the HAProxy Configuration Manual. |
| |
| |
| args [name=]<sample> ... |
| Define arguments passed into the SPOE message. |
| |
| Arguments : |
| <sample> is a sample expression. |
| |
| When the message is processed, if a sample expression is not available, it is |
| set to NULL. Arguments are processed in their declaration order and added in |
| the message in that order. It is possible to declare named arguments. |
| |
| For example: |
| args frontend=fe_id src dst |
| |
| |
| event <name> [ { if | unless } <condition> ] |
| Set the event that triggers sending of the message. It may optionally be |
| followed by an ACL-based condition, in which case it will only be evaluated |
| if the condition is true. |
| |
| ACL-based conditions are executed in the context of the stream that handle |
| the client and the server connections. |
| |
| Arguments : |
| <name> is the event name. |
| <condition> is a standard ACL-based condition. |
| |
| Supported events are: |
| - on-client-session |
| - on-server-session |
| - on-frontend-tcp-request |
| - on-backend-tcp-request |
| - on-tcp-response |
| - on-frontend-http-request |
| - on-backend-http-request |
| - on-http-response |
| |
| See section "Events & Messages" for more details about supported events. |
| See section 7 about ACL usage in the HAProxy Configuration Manual. |
| |
| 2.4. "spoe-group" section |
| -------------------------- |
| |
| This section can be used to declare a group of SPOE messages. Unlike messages |
| referenced in a "spoe-agent" section, messages inside a group are not sent on a |
| specific event. The sending must be triggered by TCP or HTTP rules, from the |
| HAProxy configuration. |
| |
| |
| spoe-group <name> |
| Create a new SPOE group with the name <name>. |
| |
| Arguments : |
| <name> is the name of the SPOE group. |
| |
| Here you define a group of SPOE messages that can be referenced in a |
| "spoe-agent" section. Following keywords are supported : |
| - messages |
| |
| See also: "spoe-agent" and "spoe-message" sections. |
| |
| |
| messages <msg-name> ... |
| Declare the list of SPOE messages belonging to the group. |
| |
| Arguments : |
| <msg-name> is the name of a SPOE message. |
| |
| Messages declared here must be found in the same engine scope, else an error |
| is triggered during the configuration parsing. Furthermore, a message belongs |
| at most to a group. You can have many "messages" lines. |
| |
| See also: "spoe-message" section. |
| |
| |
| 2.5. Example |
| ------------- |
| |
| Here is a simple but complete example that sends client-ip address to a ip |
| reputation service. This service can set the variable "ip_score" which is an |
| integer between 0 and 100, indicating its reputation (100 means totally safe |
| and 0 a blacklisted IP with no doubt). |
| |
| ### |
| ### HAProxy configuration |
| frontend www |
| mode http |
| bind *:80 |
| |
| filter spoe engine ip-reputation config spoe-ip-reputation.conf |
| |
| # Reject connection if the IP reputation is under 20 |
| tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 } |
| |
| default_backend http-servers |
| |
| backend http-servers |
| mode http |
| server http A.B.C.D:80 |
| |
| backend iprep-servers |
| mode tcp |
| balance roundrobin |
| |
| timeout connect 5s # greater than hello timeout |
| timeout server 3m # greater than idle timeout |
| |
| server iprep1 A1.B1.C1.D1:12345 |
| server iprep2 A2.B2.C2.D2:12345 |
| |
| #### |
| ### spoe-ip-reputation.conf |
| [ip-reputation] |
| |
| spoe-agent iprep-agent |
| messages get-ip-reputation |
| |
| option var-prefix iprep |
| |
| timeout hello 2s |
| timeout idle 2m |
| timeout processing 10ms |
| |
| use-backend iprep-servers |
| |
| spoe-message get-ip-reputation |
| args ip=src |
| event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } |
| |
| |
| 3. SPOP specification |
| ---------------------- |
| |
| 3.1. Data types |
| ---------------- |
| |
| Here is the bytewise representation of typed data: |
| |
| TYPED-DATA : <TYPE:4 bits><FLAGS:4 bits><DATA> |
| |
| Supported types and their representation are: |
| |
| TYPE | ID | DESCRIPTION |
| -----------------------------+-----+---------------------------------- |
| NULL | 0 | NULL : <0> |
| Boolean | 1 | BOOL : <1+FLAG> |
| 32bits signed integer | 2 | INT32 : <2><VALUE:varint> |
| 32bits unsigned integer | 3 | UINT32 : <3><VALUE:varint> |
| 64bits signed integer | 4 | INT64 : <4><VALUE:varint> |
| 32bits unsigned integer | 5 | UNIT64 : <5><VALUE:varint> |
| IPV4 | 6 | IPV4 : <6><STRUCT IN_ADDR:4 bytes> |
| IPV6 | 7 | IPV6 : <7><STRUCT IN_ADDR6:16 bytes> |
| String | 8 | STRING : <8><LENGTH:varint><BYTES> |
| Binary | 9 | BINARY : <9><LENGTH:varint><BYTES> |
| 10 -> 15 unused/reserved | - | - |
| -----------------------------+-----+---------------------------------- |
| |
| Variable-length integer (varint) are encoded using Peers encoding: |
| |
| |
| 0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ] |
| 240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ] |
| 2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ] |
| 264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ] |
| 33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ] |
| ... |
| |
| For booleans, the value (true or false) is the first bit in the FLAGS |
| bitfield. if this bit is set to 0, then the boolean is evaluated as false, |
| otherwise, the boolean is evaluated as true. |
| |
| 3.2. Frames |
| ------------ |
| |
| Exchange between HAProxy and agents are made using FRAME packets. All frames |
| must be prefixed with their size encoded on 4 bytes in network byte order: |
| |
| <FRAME-LENGTH:4 bytes> <FRAME> |
| |
| A frame always starts with its type, on one byte, followed by metadata |
| containing flags, on 4 bytes and a two variable-length integer representing the |
| stream identifier and the frame identifier inside the stream: |
| |
| FRAME : <FRAME-TYPE:1 byte> <METADATA> <FRAME-PAYLOAD> |
| METADATA : <FLAGS:4 bytes> <STREAM-ID:varint> <FRAME-ID:varint> |
| |
| Then comes the frame payload. Depending on the frame type, the payload can be |
| of three types: a simple key/value list, a list of messages or a list of |
| actions. |
| |
| FRAME-PAYLOAD : <LIST-OF-MESSAGES> | <LIST-OF-ACTIONS> | <KV-LIST> |
| |
| LIST-OF-MESSAGES : [ <MESSAGE-NAME> <NB-ARGS:1 byte> <KV-LIST> ... ] |
| MESSAGE-NAME : <STRING> |
| |
| LIST-OF-ACTIONS : [ <ACTION-TYPE:1 byte> <NB-ARGS:1 byte> <ACTION-ARGS> ... ] |
| ACTION-ARGS : [ <TYPED-DATA>... ] |
| |
| KV-LIST : [ <KV-NAME> <KV-VALUE> ... ] |
| KV-NAME : <STRING> |
| KV-VALUE : <TYPED-DATA> |
| |
| FLAGS : |
| |
| Flags are a 32 bits field. They are encoded on 4 bytes in network byte |
| order, where the bit 0 is the LSB. |
| |
| 0 1 2-31 |
| +---+---+----------+ |
| | | A | | |
| | F | B | | |
| | I | O | RESERVED | |
| | N | R | | |
| | | T | | |
| +---+---+----------+ |
| |
| FIN: Indicates that this is the final payload fragment. The first fragment |
| may also be the final fragment. |
| |
| ABORT: Indicates that the processing of the current frame must be |
| cancelled. This bit should be set on frames with a fragmented |
| payload. It can be ignore for frames with an unfragemnted |
| payload. When it is set, the FIN bit must also be set. |
| |
| |
| Frames cannot exceed a maximum size negotiated between HAProxy and agents |
| during the HELLO handshake. Most of time, payload will be small enough to send |
| it in one frame. But when supported by the peer, it will be possible to |
| fragment huge payload on many frames. This ability is announced during the |
| HELLO handshake and it can be asynmetric (supported by agents but not by |
| HAProxy or the opposite). The following rules apply to fragmentation: |
| |
| * An unfragemnted payload consists of a single frame with the FIN bit set. |
| |
| * A fragemented payload consists of several frames with the FIN bit clear and |
| terminated by a single frame with the FIN bit set. All these frames must |
| share the same STREAM-ID and FRAME-ID. The first frame must set the right |
| FRAME-TYPE (e.g, NOTIFY). The following frames must have an unset type (0). |
| |
| Beside the support of fragmented payload by a peer, some payload must not be |
| fragmented. See below for details. |
| |
| IMPORTANT : The maximum size supported by peers for a frame must be greater |
| than or equal to 256 bytes. |
| |
| 3.2.1. Frame capabilities |
| -------------------------- |
| |
| Here are the list of official capabilities that HAProxy and agents can support: |
| |
| * fragmentation: This is the ability for a peer to support fragmented |
| payload in received frames. This is an asymmectical |
| capability, it only concerns the peer that announces |
| it. This is the responsibility to the other peer to use it |
| or not. |
| |
| * pipelining: This is the ability for a peer to decouple NOTIFY and ACK |
| frames. This is a symmectical capability. To be used, it must |
| be supported by HAproxy and agents. Unlike HTTP pipelining, the |
| ACK frames can be send in any order, but always on the same TCP |
| connection used for the corresponding NOTIFY frame. |
| |
| * async: This ability is similar to the pipelining, but here any TCP |
| connection established between HAProxy and the agent can be used to |
| send ACK frames. if an agent accepts connections from multiple |
| HAProxy, it can use the "engine-id" value to group TCP |
| connections. See details about HAPROXY-HELLO frame. |
| |
| Unsupported or unknown capabilities are silently ignored, when possible. |
| |
| 3.2.2. Frame types overview |
| ---------------------------- |
| |
| Here are types of frame supported by SPOE. Frames sent by HAProxy come first, |
| then frames sent by agents : |
| |
| TYPE | ID | DESCRIPTION |
| -----------------------------+-----+------------------------------------- |
| UNSET | 0 | Used for all frames but the first when a |
| | | payload is fragmented. |
| -----------------------------+-----+------------------------------------- |
| HAPROXY-HELLO | 1 | Sent by HAProxy when it opens a |
| | | connection on an agent. |
| | | |
| HAPROXY-DISCONNECT | 2 | Sent by HAProxy when it want to close |
| | | the connection or in reply to an |
| | | AGENT-DISCONNECT frame |
| | | |
| NOTIFY | 3 | Sent by HAProxy to pass information |
| | | to an agent |
| -----------------------------+-----+------------------------------------- |
| AGENT-HELLO | 101 | Reply to a HAPROXY-HELLO frame, when |
| | | the connection is established |
| | | |
| AGENT-DISCONNECT | 102 | Sent by an agent just before closing |
| | | the connection |
| | | |
| ACK | 103 | Sent to acknowledge a NOTIFY frame |
| -----------------------------+-----+------------------------------------- |
| |
| Unknown frames may be silently skipped. |
| |
| 3.2.3. Workflow |
| ---------------- |
| |
| * Successful HELLO handshake: |
| |
| HAPROXY AGENT SRV |
| | HAPROXY-HELLO | |
| | (healthcheck: false) | |
| | --------------------------> | |
| | | |
| | AGENT-HELLO | |
| | <-------------------------- | |
| | | |
| |
| * Successful HELLO healthcheck: |
| |
| HAPROXY AGENT SRV |
| | HAPROXY-HELLO | |
| | (healthcheck: true) | |
| | --------------------------> | |
| | | |
| | AGENT-HELLO + close() | |
| | <-------------------------- | |
| | | |
| |
| |
| * Error encountered by agent during the HELLO handshake: |
| |
| HAPROXY AGENT SRV |
| | HAPROXY-HELLO | |
| | --------------------------> | |
| | | |
| | DISCONNECT + close() | |
| | <-------------------------- | |
| | | |
| |
| * Error encountered by HAProxy during the HELLO handshake: |
| |
| HAPROXY AGENT SRV |
| | HAPROXY-HELLO | |
| | --------------------------> | |
| | | |
| | AGENT-HELLO | |
| | <-------------------------- | |
| | | |
| | DISCONNECT | |
| | --------------------------> | |
| | | |
| | DISCONNECT + close() | |
| | <-------------------------- | |
| | | |
| |
| * Notify / Ack exchange (unfragmented payload): |
| |
| HAPROXY AGENT SRV |
| | NOTIFY | |
| | --------------------------> | |
| | | |
| | ACK | |
| | <-------------------------- | |
| | | |
| |
| * Notify / Ack exchange (fragmented payload): |
| |
| HAPROXY AGENT SRV |
| | NOTIFY (frag 1) | |
| | --------------------------> | |
| | | |
| | UNSET (frag 2) | |
| | --------------------------> | |
| | ... | |
| | UNSET (frag N) | |
| | --------------------------> | |
| | | |
| | ACK | |
| | <-------------------------- | |
| | | |
| |
| * Aborted fragmentation of a NOTIFY frame: |
| |
| HAPROXY AGENT SRV |
| | ... | |
| | UNSET (frag X) | |
| | --------------------------> | |
| | | |
| | ACK/ABORT | |
| | <-------------------------- | |
| | | |
| | UNSET (frag X+1) | |
| | -----------X | |
| | | |
| | | |
| |
| * Connection closed by haproxy: |
| |
| HAPROXY AGENT SRV |
| | DISCONNECT | |
| | --------------------------> | |
| | | |
| | DISCONNECT + close() | |
| | <-------------------------- | |
| | | |
| |
| * Connection closed by agent: |
| |
| HAPROXY AGENT SRV |
| | DISCONNECT + close() | |
| | <-------------------------- | |
| | | |
| |
| 3.2.4. Frame: HAPROXY-HELLO |
| ---------------------------- |
| |
| This frame is the first one exchanged between HAProxy and an agent, when the |
| connection is established. The payload of this frame is a KV-LIST. It cannot be |
| fragmented. STREAM-ID and FRAME-ID are must be set 0. |
| |
| Following items are mandatory in the KV-LIST: |
| |
| * "supported-versions" <STRING> |
| |
| Last SPOP major versions supported by HAProxy. It is a comma-separated list |
| of versions, following the format "Major.Minor". Spaces must be ignored, if |
| any. When a major version is announced by HAProxy, it means it also support |
| all previous minor versions. |
| |
| Example: "2.0, 1.5" means HAProxy supports SPOP 2.0 and 1.0 to 1.5 |
| |
| * "max-frame-size" <UINT32> |
| |
| This is the maximum size allowed for a frame. The HAPROXY-HELLO frame must |
| be lower or equal to this value. |
| |
| * "capabilities" <STRING> |
| |
| This a comma-separated list of capabilities supported by HAProxy. Spaces |
| must be ignored, if any. |
| |
| Following optional items can be added in the KV-LIST: |
| |
| * "healthcheck" <BOOLEAN> |
| |
| If this item is set to TRUE, then the HAPROXY-HELLO frame is sent during a |
| SPOE health check. When set to FALSE, this item can be ignored. |
| |
| * "engine-id" <STRING> |
| |
| This is a uniq string that identify a SPOE engine. |
| |
| To finish the HELLO handshake, the agent must return an AGENT-HELLO frame with |
| its supported SPOP version, the lower value between its maximum size allowed |
| for a frame and the HAProxy one and capabilities it supports. If an error |
| occurs or if an incompatibility is detected with the agent configuration, an |
| AGENT-DISCONNECT frame must be returned. |
| |
| 3.2.5. Frame: AGENT-HELLO |
| -------------------------- |
| |
| This frame is sent in reply to a HAPROXY-HELLO frame to finish a HELLO |
| handshake. As for HAPROXY-HELLO frame, STREAM-ID and FRAME-ID are also set |
| 0. The payload of this frame is a KV-LIST and it cannot be fragmented. |
| |
| Following items are mandatory in the KV-LIST: |
| |
| * "version" <STRING> |
| |
| This is the SPOP version the agent supports. It must follow the format |
| "Major.Minor" and it must be lower or equal than one of major versions |
| announced by HAProxy. |
| |
| * "max-frame-size" <UINT32> |
| |
| This is the maximum size allowed for a frame. It must be lower or equal to |
| the value in the HAPROXY-HELLO frame. This value will be used for all |
| subsequent frames. |
| |
| * "capabilities" <STRING> |
| |
| This a comma-separated list of capabilities supported by agent. Spaces must |
| be ignored, if any. |
| |
| At this time, if everything is ok for HAProxy (supported version and valid |
| max-frame-size value), the HELLO handshake is successfully completed. Else, |
| HAProxy sends a HAPROXY-DISCONNECT frame with the corresponding error. |
| |
| If "healthcheck" item was set to TRUE in the HAPROXY-HELLO frame, the agent can |
| safely close the connection without DISCONNECT frame. In all cases, HAProxy |
| will close the connexion at the end of the health check. |
| |
| 3.2.6. Frame: NOTIFY |
| --------------------- |
| |
| Information are sent to the agents inside NOTIFY frames. These frames are |
| attached to a stream, so STREAM-ID and FRAME-ID must be set. The payload of |
| NOTIFY frames is a LIST-OF-MESSAGES and, if supported by agents, it can be |
| fragmented. |
| |
| NOTIFY frames must be acknowledge by agents sending an ACK frame, repeating |
| right STREAM-ID and FRAME-ID. |
| |
| 3.2.7. Frame: ACK |
| ------------------ |
| |
| ACK frames must be sent by agents to reply to NOTIFY frames. STREAM-ID and |
| FRAME-ID found in a NOTIFY frame must be reuse in the corresponding ACK |
| frame. The payload of ACK frames is a LIST-OF-ACTIONS and, if supported by |
| HAProxy, it can be fragmented. |
| |
| 3.2.8. Frame: HAPROXY-DISCONNECT |
| --------------------------------- |
| |
| If an error occurs, at anytime, from the HAProxy side, a HAPROXY-DISCONNECT |
| frame is sent with information describing the error. HAProxy will wait an |
| AGENT-DISCONNECT frame in reply. All other frames will be ignored. The agent |
| must then close the socket. |
| |
| The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and |
| FRAME-ID are must be set 0. |
| |
| Following items are mandatory in the KV-LIST: |
| |
| * "status-code" <UINT32> |
| |
| This is the code corresponding to the error. |
| |
| * "message" <STRING> |
| |
| This is a textual message describing the error. |
| |
| For more information about known errors, see section "Errors & timeouts" |
| |
| 3.2.9. Frame: AGENT-DISCONNECT |
| ------------------------------- |
| |
| If an error occurs, at anytime, from the agent size, a AGENT-DISCONNECT frame |
| is sent, with information describing the error. such frame is also sent in reply |
| to a HAPROXY-DISCONNECT. The agent must close the socket just after sending |
| this frame. |
| |
| The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and |
| FRAME-ID are must be set 0. |
| |
| Following items are mandatory in the KV-LIST: |
| |
| * "status-code" <UINT32> |
| |
| This is the code corresponding to the error. |
| |
| * "message" <STRING> |
| |
| This is a textual message describing the error. |
| |
| For more information about known errors, see section "Errors & timeouts" |
| |
| 3.3. Events & Messages |
| ----------------------- |
| |
| Information about streams are sent in NOTIFY frames. You can specify which kind |
| of information to send by defining "spoe-message" sections in your SPOE |
| configuration file. for each "spoe-message" there will be a message in a NOTIFY |
| frame when the right event is triggered. |
| |
| A NOTIFY frame is sent for an specific event when there is at least one |
| "spoe-message" attached to this event. All messages for an event will be added |
| in the same NOTIFY frame. |
| |
| Here is the list of supported events: |
| |
| * on-client-session is triggered when a new client session is created. |
| This event is only available for SPOE filters |
| declared in a frontend or a listen section. |
| |
| * on-frontend-tcp-request is triggered just before the evaluation of |
| "tcp-request content" rules on the frontend side. |
| This event is only available for SPOE filters |
| declared in a frontend or a listen section. |
| |
| * on-backend-tcp-request is triggered just before the evaluation of |
| "tcp-request content" rules on the backend side. |
| This event is skipped for SPOE filters declared |
| in a listen section. |
| |
| * on-frontend-http-request is triggered just before the evaluation of |
| "http-request" rules on the frontend side. This |
| event is only available for SPOE filters declared |
| in a frontend or a listen section. |
| |
| * on-backend-http-request is triggered just before the evaluation of |
| "http-request" rules on the backend side. This |
| event is skipped for SPOE filters declared in a |
| listen section. |
| |
| * on-server-session is triggered when the session with the server is |
| established. |
| |
| * on-tcp-response is triggered just before the evaluation of |
| "tcp-response content" rules. |
| |
| * on-http-response is triggered just before the evaluation of |
| "http-response" rules. |
| |
| |
| The stream processing will loop on these events, when triggered, waiting the |
| agent reply. |
| |
| 3.4. Actions |
| ------------- |
| |
| An agent must acknowledge each NOTIFY frame by sending the corresponding ACK |
| frame. Actions can be added in these frames to dynamically take action on the |
| processing of a stream. |
| |
| Here is the list of supported actions: |
| |
| * set-var set the value for an existing variable. 3 arguments must be |
| attached to this action: the variable scope (proc, sess, txn, |
| req or res), the variable name (a string) and its value. |
| |
| ACTION-SET-VAR : <SET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME><VAR-VALUE> |
| |
| SET-VAR : <1> |
| NB-ARGS : <3> |
| VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE> |
| VAR-NAME : <STRING> |
| VAR-VALUE : <TYPED-DATA> |
| |
| PROCESS : <0> |
| SESSION : <1> |
| TRANSACTION : <2> |
| REQUEST : <3> |
| RESPONSE : <4> |
| |
| * unset-var unset the value for an existing variable. 2 arguments must be |
| attached to this action: the variable scope (proc, sess, txn, |
| req or res) and the variable name (a string). |
| |
| ACTION-UNSET-VAR : <UNSET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME> |
| |
| UNSET-VAR : <2> |
| NB-ARGS : <2> |
| VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE> |
| VAR-NAME : <STRING> |
| |
| PROCESS : <0> |
| SESSION : <1> |
| TRANSACTION : <2> |
| REQUEST : <3> |
| RESPONSE : <4> |
| |
| |
| NOTE: Name of the variables will be automatically prefixed by HAProxy to avoid |
| name clashes with other variables used in HAProxy. Moreover, unknown |
| variable will be silently ignored. |
| |
| 3.5. Errors & timeouts |
| ---------------------- |
| |
| Here is the list of all known errors: |
| |
| STATUS CODE | DESCRIPTION |
| ----------------+-------------------------------------------------------- |
| 0 | normal (no error occurred) |
| 1 | I/O error |
| 2 | A timeout occurred |
| 3 | frame is too big |
| 4 | invalid frame received |
| 5 | version value not found |
| 6 | max-frame-size value not found |
| 7 | capabilities value not found |
| 8 | unsupported version |
| 9 | max-frame-size too big or too small |
| 10 | payload fragmentation is not supported |
| 11 | invalid interlaced frames |
| 12 | frame-id not found (it does not match any referenced frame) |
| 13 | resource allocation error |
| 99 | an unknown error occurrde |
| ----------------+-------------------------------------------------------- |
| |
| An agent can define its own errors using a not yet assigned status code. |
| |
| IMPORTANT NOTE: By default, for a specific stream, when an abnormal/unexpected |
| error occurs, the SPOE is disabled for all the transaction. So |
| if you have several events configured, such error on an event |
| will disabled all followings. For TCP streams, this will |
| disable the SPOE for the whole session. For HTTP streams, this |
| will disable it for the transaction (request and response). |
| See 'option continue-on-error' to bypass this limitation. |
| |
| To avoid a stream to wait undefinetly, you must carefully choose the |
| acknowledgement timeout. In most of cases, it will be quiet low. But it depends |
| on the responsivness of your service. |
| |
| You must also choose idle timeout carefully. Because connection with your |
| service depends on the backend configuration used by the SPOA, it is important |
| to use a lower value for idle timeout than the server timeout. Else the |
| connection will be closed by HAProxy. The same is true for hello timeout. You |
| should choose a lower value than the connect timeout. |
| |
| 4. Logging |
| ----------- |
| |
| Activity of an SPOE is logged using HAProxy's logger. The messages are logged |
| in the context of the streams that handle the client and the server |
| connections. A message is emitted for each event or group handled by an |
| SPOE. Depending on the status code, the log level will be different. In the |
| normal case, when no error occurred, the message is logged with the level |
| LOG_NOTICE. Otherwise, the message is logged with the level LOG_WARNING. |
| |
| The messages are logged using the agent's logger, if defined, and use the |
| following format: |
| |
| SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUS-CODE reqT/qT/wT/resT/pT \ |
| <idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed> |
| |
| AGENT is the agent name |
| TYPE is EVENT of GROUP |
| NAME is the event or the group name |
| STREAM-ID is an integer, the unique id of the stream |
| STATUS_CODE is the processing's status code |
| reqT/qT/wT/resT/pT are the following time events: |
| |
| * reqT : the encoding time. It includes ACLs processing, if any. For |
| fragmented frames, it is the sum of all fragments. |
| * qT : the delay before the request gets out the sending queue. For |
| fragmented frames, it is the sum of all fragments. |
| * wT : the delay before the response is received. No fragmentation |
| supported here. |
| * resT : the delay to process the response. No fragmentation supported |
| here. |
| * pT : the delay to process the event or the group. From the stream |
| point of view, it is the latency added by the SPOE processing. |
| It is more or less the sum of values above. |
| |
| <idle> is the numbers of idle SPOE applets |
| <applets> is the numbers of SPOE applets |
| <nb_sending> is the numbers of streams waiting to send data |
| <nb_waiting> is the numbers of streams waiting for a ack |
| <nb_error> is the numbers of processing errors |
| <nb_processed> is the numbers of events/groups processed |
| |
| |
| For all these time events, -1 means the processing was interrupted before the |
| end. So -1 for the queue time means the request was never dequeued. For |
| fragmented frames it is harder to know when the interruption happened. |
| |
| /* |
| * Local variables: |
| * fill-column: 79 |
| * End: |
| */ |