Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 1 | ----------------------------------------------- |
| 2 | Stream Processing Offload Engine (SPOE) |
| 3 | Version 1.0 |
| 4 | ( Last update: 2016-11-07 ) |
| 5 | ----------------------------------------------- |
| 6 | Author : Christopher Faulet |
| 7 | Contact : cfaulet at haproxy dot com |
| 8 | |
| 9 | |
| 10 | SUMMARY |
| 11 | -------- |
| 12 | |
| 13 | 0. Terms |
| 14 | 1. Introduction |
| 15 | 2. SPOE configuration |
| 16 | 2.1. SPOE scope |
| 17 | 2.2. "spoe-agent" section |
| 18 | 2.3. "spoe-message" section |
| 19 | 2.4. Example |
| 20 | 3. SPOP specification |
| 21 | 3.1. Data types |
| 22 | 3.2. Frames |
| 23 | 3.2.1. Frame capabilities |
| 24 | 3.2.2. Frame types overview |
| 25 | 3.2.3. Workflow |
| 26 | 3.2.4. Frame: HAPROXY-HELLO |
| 27 | 3.2.5. Frame: AGENT-HELLO |
| 28 | 3.2.6. Frame: NOTIFY |
| 29 | 3.2.7. Frame: ACK |
| 30 | 3.2.8. Frame: HAPROXY-DISCONNECT |
| 31 | 3.2.9. Frame: AGENT-DISCONNECT |
| 32 | 3.3. Events & messages |
| 33 | 3.4. Actions |
| 34 | 3.5. Error & timeouts |
| 35 | |
| 36 | |
| 37 | 0. Terms |
| 38 | --------- |
| 39 | |
| 40 | * SPOE : Stream Processing Offload Engine. |
| 41 | |
| 42 | A SPOE is a filter talking to servers managed ba a SPOA to offload the |
| 43 | stream processing. An engine is attached to a proxy. A proxy can have |
| 44 | several engine. Each engine is linked to an agent and only one. |
| 45 | |
| 46 | * SPOA : Stream Processing Offload Agent. |
| 47 | |
| 48 | A SPOA is a service that will receive info from a SPOE to offload the |
| 49 | stream processing. An agent manages several servers. It uses a backend to |
| 50 | reference all of them. By extension, these servers can also be called |
| 51 | agents. |
| 52 | |
| 53 | * SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA |
| 54 | servers. |
| 55 | |
| 56 | This protocol is used by engines to talk to agents. It is an in-house |
| 57 | binary protocol described in this documentation. |
| 58 | |
| 59 | |
| 60 | 1. Introduction |
| 61 | ---------------- |
| 62 | |
| 63 | SPOE is a feature introduced in HAProxy 1.7. It makes possible the |
| 64 | communication with external components to retrieve some info. The idea started |
| 65 | with the problems caused by most ldap libs not working fine in event-driven |
| 66 | systems (often at least the connect() is blocking). So, it is hard to properly |
| 67 | implement Single Sign On solution (SSO) in HAProxy. The SPOE will ease this |
| 68 | kind of processing, or we hope so. |
| 69 | |
| 70 | Now, the aim of SPOE is to allow any kind of offloading on the streams. First |
| 71 | releases, besides being experimental, won't do lot of things. As we will see, |
| 72 | there are few handled events and even less actions supported. Actually, for |
| 73 | now, the SPOE can offload the processing before "tcp-request content", |
| 74 | "tcp-response content", "http-request" and "http-response" rules. And it only |
| 75 | supports variables definition. But, in spite of these limited features, we can |
| 76 | easily imagine to implement SSO solution, ip reputation or ip geolocation |
| 77 | services. |
| 78 | |
| 79 | |
| 80 | 2. SPOE configuration |
| 81 | ---------------------- |
| 82 | |
| 83 | Because SPOE is implemented as a filter, To use it, you must declare a "filter |
| 84 | spoe" line in a proxy section (frontend/backend/listen) : |
| 85 | |
| 86 | frontend my-front |
| 87 | ... |
| 88 | filter spoe [engine <name>] config <file> |
| 89 | ... |
| 90 | |
| 91 | The "config" parameter is mandatory. It specififies the SPOE configuration |
| 92 | file. The engine name is optional. It can be set to declare the scope to use in |
| 93 | the SPOE configuration. So it is possible to use the same SPOE configuration |
| 94 | for several engines. If no name is provided, the SPOE configuration must not |
| 95 | contain any scope directive. |
| 96 | |
| 97 | We use a separate configuration file on purpose. By commenting SPOE filter |
| 98 | line, you completly disable the feature, including the parsing of sections |
| 99 | reserved to SPOE. This is also a way to keep the HAProxy configuration clean. |
| 100 | |
| 101 | A SPOE configuration file must contains, at least, the SPOA configuration |
| 102 | ("spoe-agent" section) and SPOE messages ("spoe-message" section) attached to |
| 103 | this agent. Unused messages (not reference in "spoe-agent" section) will be |
| 104 | ignored. |
| 105 | |
| 106 | IMPORTANT : The configuration of a SPOE filter must be located in a dedicated |
| 107 | file. But the backend used by a SPOA must be declared in HAProxy configuration |
| 108 | file. |
| 109 | |
| 110 | 2.1. SPOE scope |
| 111 | ------------------------- |
| 112 | |
| 113 | If you specify an engine name on the SPOE filter line, then you need to define |
| 114 | scope in the SPOE configuration with the same name. You can have several SPOE |
| 115 | scope in the same file. In each scope, you must define one and only one |
| 116 | "spoe-agent" section to configure the SPOA linked to your SPOE and several |
| 117 | "spoe-message" sections to describe messages sent to servers mananger by your |
| 118 | SPOA. |
| 119 | |
| 120 | A SPOE scope starts with this kind of line : |
| 121 | |
| 122 | [<name>] |
| 123 | |
| 124 | where <name> is the same engine name specified on the SPOE filter line. The |
| 125 | scope ends when the file ends or when another scope is found. |
| 126 | |
| 127 | Example : |
| 128 | [my-first-engine] |
| 129 | spoe-agent my-agent |
| 130 | ... |
| 131 | spoe-message msg1 |
| 132 | ... |
| 133 | spoe-message msg2 |
| 134 | ... |
| 135 | |
| 136 | [my-second-engine] |
| 137 | ... |
| 138 | |
| 139 | If no engine name is provided on the SPOE filter line, no SPOE scope must be |
| 140 | found in the SPOE configuration file. All the file is considered to be in the |
| 141 | same anonymous and implicit scope. |
| 142 | |
| 143 | 2.2. "spoe-agent" section |
| 144 | -------------------------- |
| 145 | |
| 146 | For each engine, you must define one and only one "spoe-agent" section. In this |
| 147 | section, you will declare SPOE messages and the backend you will use. You will |
| 148 | also set timeouts and options to customize your agent's behaviour. |
| 149 | |
| 150 | |
| 151 | spoe-agent <name> |
| 152 | Create a new SPOA with the name <name>. It must have one and only one |
| 153 | "spoe-agent" definition by SPOE scope. |
| 154 | |
| 155 | Arguments : |
| 156 | <name> is the name of the agent section. |
| 157 | |
| 158 | following keywords are supported : |
Christopher Faulet | 4802672 | 2016-11-16 15:01:12 +0100 | [diff] [blame] | 159 | - maxconnrate |
| 160 | - maxerrrate |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 161 | - messages |
Christopher Faulet | ea62c2a | 2016-11-14 10:54:21 +0100 | [diff] [blame] | 162 | - option continue-on-error |
Christopher Faulet | 985532d | 2016-11-16 15:36:19 +0100 | [diff] [blame] | 163 | - option set-on-error |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 164 | - option var-prefix |
Christopher Faulet | 03a3449 | 2016-11-19 16:47:56 +0100 | [diff] [blame] | 165 | - timeout hello|idle|processing |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 166 | - use-backend |
| 167 | |
| 168 | |
Christopher Faulet | 4802672 | 2016-11-16 15:01:12 +0100 | [diff] [blame] | 169 | maxconnrate <number> |
| 170 | Set the maximum number of connections per second to <number>. The SPOE will |
| 171 | stop to open new connections if the maximum is reached and will wait to |
| 172 | acquire an existing one. So it is important to set "timeout hello" to a |
| 173 | relatively small value. |
| 174 | |
| 175 | |
| 176 | maxerrrate <number> |
| 177 | Set the maximum number of errors per second to <number>. The SPOE will stop |
| 178 | its processing if the maximum is reached. |
| 179 | |
| 180 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 181 | messages <msg-name> ... |
| 182 | Declare the list of SPOE messages that an agent will handle. |
| 183 | |
| 184 | Arguments : |
| 185 | <msg-name> is the name of a SPOE message. |
| 186 | |
| 187 | Messages declared here must be found in the same engine scope, else an error |
| 188 | is triggered during the configuration parsing. You can have many "messages" |
| 189 | lines. |
| 190 | |
| 191 | See also: "spoe-message" section. |
| 192 | |
| 193 | |
Christopher Faulet | ea62c2a | 2016-11-14 10:54:21 +0100 | [diff] [blame] | 194 | option continue-on-error |
| 195 | Do not stop the events processing when an error occurred on a stream. |
| 196 | |
| 197 | By default, for a specific stream, when an abnormal/unexpected error occurs, |
| 198 | the SPOE is disabled for all the transaction. So if you have several events |
| 199 | configured, such error on an event will disabled all followings. For TCP |
| 200 | streams, this will disable the SPOE for the whole session. For HTTP streams, |
| 201 | this will disable it for the transaction (request and response). |
| 202 | |
| 203 | When set, this option bypass this behaviour and only the current event will |
| 204 | be ignored. |
| 205 | |
| 206 | |
Christopher Faulet | 985532d | 2016-11-16 15:36:19 +0100 | [diff] [blame] | 207 | option set-on-error <var name> |
| 208 | Define the variable to set when an error occurred during an event processing. |
| 209 | |
| 210 | Arguments : |
| 211 | |
| 212 | <var name> is the variable name, without the scope. The name may only |
| 213 | contain characters 'a-z', 'A-Z', '0-9', '.' and '_'. |
| 214 | |
| 215 | This variable will only be set when an error occurred in the scope of the |
| 216 | transaction. As for all other variables define by the SPOE, it will be |
| 217 | prefixed. So, if your variable name is "error" and your prefix is |
| 218 | "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". |
| 219 | |
| 220 | When set, the variable is the boolean "true". Note that if "option |
| 221 | continue-on-error" is set, the variable is not automatically removed between |
| 222 | events processing. |
| 223 | |
| 224 | See also: "option continue-on-error", "option var-prefix". |
| 225 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 226 | option var-prefix <prefix> |
| 227 | Define the prefix used when variables are set by an agent. |
| 228 | |
| 229 | Arguments : |
| 230 | |
| 231 | <prefix> is the prefix used to limit the scope of variables set by an |
| 232 | agent. |
| 233 | |
| 234 | To avoid conflict with other variables defined by HAProxy, all variables |
| 235 | names will be prefixed. By default, the "spoe-agent" name is used. This |
| 236 | option can be used to customize it. |
| 237 | |
| 238 | The prefix will be added between the variable scope and its name, separated |
| 239 | by a '.'. It may only contain characters 'a-z', 'A-Z', '0-9', '.' and '_', as |
| 240 | for variables name. In HAProxy configuration, you need to use this prefix as |
| 241 | a part of the variables name. For example, if an agent define the variable |
| 242 | "myvar" in the "txn" scope, with the prefix "my_spoe_pfx", then you should |
| 243 | use "txn.my_spoe_pfx.myvar" name in your HAProxy configuration. |
| 244 | |
| 245 | An agent will never set new variables at runtime. It can only set new value |
| 246 | for existing ones. |
| 247 | |
| 248 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 249 | timeout hello <timeout> |
| 250 | Set the maximum time to wait for an agent to receive the AGENT-HELLO frame. |
Christopher Faulet | f7a3092 | 2016-11-10 15:04:51 +0100 | [diff] [blame] | 251 | It is applied on the stream that handle the connection with the agent. |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 252 | |
| 253 | Arguments : |
| 254 | <timeout> is the timeout value specified in milliseconds by default, but |
| 255 | can be in any other unit if the number is suffixed by the unit, |
| 256 | as explained at the top of this document. |
| 257 | |
| 258 | This timeout is an applicative timeout. It differ from "timeout connect" |
| 259 | defined on backends. |
| 260 | |
| 261 | |
| 262 | timeout idle <timeout> |
Christopher Faulet | f7a3092 | 2016-11-10 15:04:51 +0100 | [diff] [blame] | 263 | Set the maximum time to wait for an agent to close an idle connection. It is |
| 264 | applied on the stream that handle the connection with the agent. |
| 265 | |
| 266 | Arguments : |
| 267 | <timeout> is the timeout value specified in milliseconds by default, but |
| 268 | can be in any other unit if the number is suffixed by the unit, |
| 269 | as explained at the top of this document. |
| 270 | |
| 271 | |
| 272 | timeout processing <timeout> |
| 273 | Set the maximum time to wait for a stream to process an event, i.e to acquire |
| 274 | a stream to talk with an agent, to encode all messages, to send the NOTIFY |
| 275 | frame, to receive the corrsponding acknowledgement and to process all |
| 276 | actions. It is applied on the stream that handle the client and the server |
| 277 | sessions. |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 278 | |
| 279 | Arguments : |
| 280 | <timeout> is the timeout value specified in milliseconds by default, but |
| 281 | can be in any other unit if the number is suffixed by the unit, |
| 282 | as explained at the top of this document. |
| 283 | |
| 284 | |
| 285 | use-backend <backend> |
| 286 | Specify the backend to use. It must be defined. |
| 287 | |
| 288 | Arguments : |
| 289 | <backend> is the name of a valid "backend" section. |
| 290 | |
| 291 | |
| 292 | 2.3. "spoe-message" section |
| 293 | ---------------------------- |
| 294 | |
| 295 | To offload the stream processing, SPOE will send messages with specific |
| 296 | information at a specific moment in the stream life and will wait for |
| 297 | corresponding replies to know what to do. |
| 298 | |
| 299 | |
| 300 | spoe-message <name> |
| 301 | Create a new SPOE message with the name <name>. |
| 302 | |
| 303 | Arguments : |
| 304 | <name> is the name of the SPOE message. |
| 305 | |
| 306 | Here you define a message that can be referenced in a "spoe-agent" |
| 307 | section. Following keywords are supported : |
| 308 | - args |
| 309 | - event |
| 310 | |
| 311 | See also: "spoe-agent" section. |
| 312 | |
| 313 | |
| 314 | args [name=]<sample> ... |
| 315 | Define arguments passed into the SPOE message. |
| 316 | |
| 317 | Arguments : |
| 318 | <sample> is a sample expression. |
| 319 | |
| 320 | When the message is processed, if a sample expression is not available, it is |
| 321 | set to NULL. Arguments are processed in their declaration order and added in |
| 322 | the message in that order. It is possible to declare named arguements. |
| 323 | |
| 324 | For example: |
| 325 | args frontend=fe_id src dst |
| 326 | |
| 327 | |
| 328 | event <name> |
| 329 | Set the event that triggers sending of the message. |
| 330 | |
| 331 | Argument : |
| 332 | <name> is the event name. |
| 333 | |
| 334 | Supported events are: |
| 335 | - on-client-session |
Christopher Faulet | 1002aac | 2016-12-09 17:41:54 +0100 | [diff] [blame] | 336 | - on-server-session |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 337 | - on-frontend-tcp-request |
| 338 | - on-backend-tcp-request |
| 339 | - on-tcp-response |
| 340 | - on-frontend-http-request |
| 341 | - on-backend-http-request |
| 342 | - on-http-response |
| 343 | |
| 344 | See section 3.5 about Events. |
| 345 | |
| 346 | 2.4. Example |
| 347 | ------------- |
| 348 | |
| 349 | Here is a simple but complete example that sends client-ip address to a ip |
| 350 | reputation service. This service can set the variable "ip_score" which is an |
| 351 | integer between 0 and 100, indicating its reputation (100 means totally safe |
| 352 | and 0 a blacklisted IP with no doubt). |
| 353 | |
| 354 | ### |
| 355 | ### HAProxy configuration |
| 356 | frontend www |
| 357 | mode http |
| 358 | bind *:80 |
| 359 | |
| 360 | filter spoe engine ip-reputation config spoe-ip-reputation.conf |
| 361 | |
| 362 | # Reject connection if the IP reputation is under 20 |
| 363 | tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 } |
| 364 | |
| 365 | default_backend http-servers |
| 366 | |
| 367 | backend http-servers |
| 368 | mode http |
| 369 | server http A.B.C.D:80 |
| 370 | |
| 371 | backend iprep-servers |
| 372 | mode tcp |
| 373 | balance roundrobin |
| 374 | |
| 375 | timeout connect 5s # greater than hello timeout |
| 376 | timeout server 3m # greater than idle timeout |
| 377 | |
| 378 | server iprep1 A1.B1.C1.D1:12345 |
| 379 | server iprep2 A2.B2.C2.D2:12345 |
| 380 | |
| 381 | #### |
| 382 | ### spoe-ip-reputation.conf |
| 383 | [ip-reputation] |
| 384 | |
| 385 | spoe-agent iprep-agent |
| 386 | messages get-ip-reputation |
| 387 | |
| 388 | option var-prefix iprep |
| 389 | |
Christopher Faulet | 03a3449 | 2016-11-19 16:47:56 +0100 | [diff] [blame] | 390 | timeout hello 2s |
| 391 | timeout idle 2m |
| 392 | timeout processing 10ms |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 393 | |
| 394 | use-backend iprep-servers |
| 395 | |
| 396 | spoe-message get-ip-reputation |
| 397 | args ip=src |
| 398 | event on-client-session |
| 399 | |
| 400 | |
| 401 | 3. SPOP specification |
| 402 | ---------------------- |
| 403 | |
| 404 | 3.1. Data types |
| 405 | ---------------- |
| 406 | |
| 407 | Here is the bytewise representation of typed data: |
| 408 | |
| 409 | TYPED-DATA : <TYPE:4 bits><FLAGS:4 bits><DATA> |
| 410 | |
| 411 | Supported types and their representation are: |
| 412 | |
| 413 | TYPE | ID | DESCRIPTION |
| 414 | -----------------------------+-----+---------------------------------- |
| 415 | NULL | 0 | NULL : <0> |
| 416 | Boolean | 1 | BOOL : <1+FLAG> |
| 417 | 32bits signed integer | 2 | INT32 : <2><VALUE:varint> |
| 418 | 32bits unsigned integer | 3 | UINT32 : <3><VALUE:varint> |
| 419 | 64bits signed integer | 4 | INT64 : <4><VALUE:varint> |
| 420 | 32bits unsigned integer | 5 | UNIT64 : <5><VALUE:varint> |
| 421 | IPV4 | 6 | IPV4 : <6><STRUCT IN_ADDR:4 bytes> |
| 422 | IPV6 | 7 | IPV6 : <7><STRUCT IN_ADDR6:16 bytes> |
| 423 | String | 8 | STRING : <8><LENGTH:varint><BYTES> |
| 424 | Binary | 9 | BINARY : <9><LENGTH:varint><BYTES> |
| 425 | 10 -> 15 unused/reserved | - | - |
| 426 | -----------------------------+-----+---------------------------------- |
| 427 | |
| 428 | Variable-length integer (varint) are encoded using Peers encoding: |
| 429 | |
| 430 | |
| 431 | 0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ] |
| 432 | 240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ] |
| 433 | 2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ] |
| 434 | 264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ] |
| 435 | 33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ] |
| 436 | ... |
| 437 | |
| 438 | For booleans, the value (true or false) is the first bit in the FLAGS |
| 439 | bitfield. if this bit is set to 0, then the boolean is evaluated as false, |
| 440 | otherwise, the boolean is evaluated as true. |
| 441 | |
| 442 | 3.2. Frames |
| 443 | ------------ |
| 444 | |
| 445 | Exchange between HAProxy and agents are made using FRAME packets. All frames |
| 446 | must be prefixed with their size encoded on 4 bytes in network byte order: |
| 447 | |
| 448 | <FRAME-LENGTH:4 bytes> <FRAME> |
| 449 | |
| 450 | A frame always starts with its type, on one byte, followed by metadata |
| 451 | containing flags, on 4 bytes and a two variable-length integer representing the |
| 452 | stream identifier and the frame identifier inside the stream: |
| 453 | |
| 454 | FRAME : <FRAME-TYPE:1 byte> <METADATA> <FRAME-PAYLOAD> |
| 455 | METADATA : <FLAGS:4 bytes> <STREAM-ID:varint> <FRAME-ID:varint> |
| 456 | |
| 457 | Then comes the frame payload. Depending on the frame type, the payload can be |
| 458 | of three types: a simple key/value list, a list of messages or a list of |
| 459 | actions. |
| 460 | |
| 461 | FRAME-PAYLOAD : <LIST-OF-MESSAGES> | <LIST-OF-ACTIONS> | <KV-LIST> |
| 462 | |
| 463 | LIST-OF-MESSAGES : [ <MESSAGE-NAME> <NB-ARGS:1 byte> <KV-LIST> ... ] |
| 464 | MESSAGE-NAME : <STRING> |
| 465 | |
| 466 | LIST-OF-ACTIONS : [ <ACTION-TYPE:1 byte> <NB-ARGS:1 byte> <ACTION-ARGS> ... ] |
| 467 | ACTION-ARGS : [ <TYPED-DATA>... ] |
| 468 | |
| 469 | KV-LIST : [ <KV-NAME> <KV-VALUE> ... ] |
| 470 | KV-NAME : <STRING> |
| 471 | KV-VALUE : <TYPED-DATA> |
| 472 | |
| 473 | FLAGS : 0 1-31 |
| 474 | +---+-----------+ |
| 475 | | F| | |
| 476 | | I| RESERVED | |
| 477 | | N| | |
| 478 | +--+------------+ |
| 479 | |
| 480 | FIN: Indicates that this is the final payload fragment. The first fragment |
| 481 | may also be the final fragment. |
| 482 | |
| 483 | Frames cannot exceed a maximum size negociated between HAProxy and agents |
| 484 | during the HELLO handshake. Most of time, payload will be small enough to send |
| 485 | it in one frame. But when supported by the peer, it will be possible to |
| 486 | fragment huge payload on many frames. This ability is announced during the |
| 487 | HELLO handshake and it can be asynmetric (supported by agents but not by |
| 488 | HAProxy or the opposite). The following rules apply to fragmentation: |
| 489 | |
| 490 | * An unfragemnted payload consists of a single frame with the FIN bit set. |
| 491 | |
| 492 | * A fragemented payload consists of several frames with the FIN bit clear and |
| 493 | terminated by a single frame with the FIN bit set. All these frames must |
| 494 | share the same STREAM-ID and FRAME-ID. And, of course, the FRAME-TYPE must |
| 495 | be the same. |
| 496 | |
| 497 | Beside the support of fragmented payload by a peer, some payload must not be |
| 498 | fragmented. See below for details. |
| 499 | |
| 500 | IMPORTANT : The maximum size supported by peers for a frame must be greater or |
| 501 | equal to 256 bytes. |
| 502 | |
| 503 | 3.2.1. Frame capabilities |
| 504 | -------------------------- |
| 505 | |
| 506 | Here are the list of official capabilities that HAProxy and agents can support: |
| 507 | |
| 508 | * fragmentation: This is the abaility for a peer to support fragmented |
| 509 | payload in received frames. |
| 510 | |
| 511 | Unsupported or unknown capabilities are silently ignored, when possible. |
| 512 | |
| 513 | 3.2.2. Frame types overview |
| 514 | ---------------------------- |
| 515 | |
| 516 | Here are types of frame supported by SPOE. Frames sent by HAProxy come first, |
| 517 | then frames sent by agents : |
| 518 | |
| 519 | TYPE | ID | DESCRIPTION |
| 520 | -----------------------------+-----+------------------------------------- |
| 521 | HAPROXY-HELLO | 1 | Sent by HAProxy when it opens a |
| 522 | | | connection on an agent. |
| 523 | | | |
| 524 | HAPROXY-DISCONNECT | 2 | Sent by HAProxy when it want to close |
| 525 | | | the connection or in reply to an |
| 526 | | | AGENT-DISCONNECT frame |
| 527 | | | |
| 528 | NOTIFY | 3 | Sent by HAProxy to pass information |
| 529 | | | to an agent |
| 530 | -----------------------------+-----+------------------------------------- |
| 531 | AGENT-HELLO | 101 | Reply to a HAPROXY-HELLO frame, when |
| 532 | | | the connection is established |
| 533 | | | |
| 534 | AGENT-DISCONNECT | 102 | Sent by an agent just before closing |
| 535 | | | the connection |
| 536 | | | |
| 537 | ACK | 103 | Sent to acknowledge a NOTIFY frame |
| 538 | -----------------------------+-----+------------------------------------- |
| 539 | |
| 540 | Unknown frames may be silently skipped. |
| 541 | |
| 542 | 3.2.3. Workflow |
| 543 | ---------------- |
| 544 | |
| 545 | * Successful HELLO handshake: |
| 546 | |
| 547 | HAPROXY AGENT SRV |
| 548 | | HAPROXY-HELLO | |
Christopher Faulet | ba7bc16 | 2016-11-07 21:07:38 +0100 | [diff] [blame] | 549 | | (healthcheck: false) | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 550 | | --------------------------> | |
| 551 | | | |
| 552 | | AGENT-HELLO | |
| 553 | | <-------------------------- | |
| 554 | | | |
| 555 | |
Christopher Faulet | ba7bc16 | 2016-11-07 21:07:38 +0100 | [diff] [blame] | 556 | * Successful HELLO healthcheck: |
| 557 | |
| 558 | HAPROXY AGENT SRV |
| 559 | | HAPROXY-HELLO | |
| 560 | | (healthcheck: true) | |
| 561 | | --------------------------> | |
| 562 | | | |
| 563 | | AGENT-HELLO + close() | |
| 564 | | <-------------------------- | |
| 565 | | | |
| 566 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 567 | |
| 568 | * Error encountered by agent during the HELLO handshake: |
| 569 | |
| 570 | HAPROXY AGENT SRV |
| 571 | | HAPROXY-HELLO | |
| 572 | | --------------------------> | |
| 573 | | | |
| 574 | | DISCONNECT + close() | |
| 575 | | <-------------------------- | |
| 576 | | | |
| 577 | |
| 578 | * Error encountered by HAProxy during the HELLO handshake: |
| 579 | |
| 580 | HAPROXY AGENT SRV |
| 581 | | HAPROXY-HELLO | |
| 582 | | --------------------------> | |
| 583 | | | |
| 584 | | AGENT-HELLO | |
| 585 | | <-------------------------- | |
| 586 | | | |
| 587 | | DISCONNECT | |
| 588 | | --------------------------> | |
| 589 | | | |
| 590 | | DISCONNECT + close() | |
| 591 | | <-------------------------- | |
| 592 | | | |
| 593 | |
| 594 | * Notify / Ack exchange: |
| 595 | |
| 596 | HAPROXY AGENT SRV |
| 597 | | NOTIFY | |
| 598 | | --------------------------> | |
| 599 | | | |
| 600 | | ACK | |
| 601 | | <-------------------------- | |
| 602 | | | |
| 603 | |
| 604 | * Connection closed by haproxy: |
| 605 | |
| 606 | HAPROXY AGENT SRV |
| 607 | | DISCONNECT | |
| 608 | | --------------------------> | |
| 609 | | | |
| 610 | | DISCONNECT + close() | |
| 611 | | <-------------------------- | |
| 612 | | | |
| 613 | |
| 614 | * Connection closed by agent: |
| 615 | |
| 616 | HAPROXY AGENT SRV |
| 617 | | DISCONNECT + close() | |
| 618 | | <-------------------------- | |
| 619 | | | |
| 620 | |
| 621 | 3.2.4. Frame: HAPROXY-HELLO |
| 622 | ---------------------------- |
| 623 | |
| 624 | This frame is the first one exchanged between HAProxy and an agent, when the |
| 625 | connection is established. The payload of this frame is a KV-LIST. It cannot be |
| 626 | fragmented. STREAM-ID and FRAME-ID are must be set 0. |
| 627 | |
| 628 | Following items are mandatory in the KV-LIST: |
| 629 | |
| 630 | * "supported-versions" <STRING> |
| 631 | |
| 632 | Last SPOP major versions supported by HAProxy. It is a comma-separated list |
| 633 | of versions, following the format "Major.Minor". Spaces must be ignored, if |
| 634 | any. When a major version is announced by HAProxy, it means it also support |
| 635 | all previous minor versions. |
| 636 | |
| 637 | Example: "2.0, 1.5" means HAProxy supports SPOP 2.0 and 1.0 to 1.5 |
| 638 | |
| 639 | * "max-frame-size" <UINT32> |
| 640 | |
| 641 | This is the maximum size allowed for a frame. The HAPROXY-HELLO frame must |
| 642 | be lower or equal to this value. |
| 643 | |
| 644 | * "capabilities" <STRING> |
| 645 | |
| 646 | This a comma-separated list of capabilities supported by HAProxy. Spaces |
| 647 | must be ignored, if any. |
| 648 | |
Christopher Faulet | ba7bc16 | 2016-11-07 21:07:38 +0100 | [diff] [blame] | 649 | Following optional items can be added in the KV-LIST: |
| 650 | |
| 651 | * "healthcheck" <BOOLEAN> |
| 652 | |
| 653 | If this item is set to TRUE, then the HAPROXY-HELLO frame is sent during a |
| 654 | SPOE health check. When set to FALSE, this item can be ignored. |
| 655 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 656 | To finish the HELLO handshake, the agent must return an AGENT-HELLO frame with |
| 657 | its supported SPOP version, the lower value between its maximum size allowed |
| 658 | for a frame and the HAProxy one and capabilities it supports. If an error |
| 659 | occurs or if an incompatibility is detected with the agent configuration, an |
| 660 | AGENT-DISCONNECT frame must be returned. |
| 661 | |
| 662 | 3.2.5. Frame: AGENT-HELLO |
| 663 | -------------------------- |
| 664 | |
| 665 | This frame is sent in reply to a HAPROXY-HELLO frame to finish a HELLO |
| 666 | handshake. As for HAPROXY-HELLO frame, STREAM-ID and FRAME-ID are also set |
| 667 | 0. The payload of this frame is a KV-LIST and it cannot be fragmented. |
| 668 | |
| 669 | Following items are mandatory in the KV-LIST: |
| 670 | |
| 671 | * "version" <STRING> |
| 672 | |
| 673 | This is the SPOP version the agent supports. It must follow the format |
| 674 | "Major.Minor" and it must be lower or equal than one of major versions |
| 675 | announced by HAProxy. |
| 676 | |
| 677 | * "max-frame-size" <UINT32> |
| 678 | |
| 679 | This is the maximum size allowed for a frame. It must be lower or equal to |
| 680 | the value in the HAPROXY-HELLO frame. This value will be used for all |
| 681 | subsequent frames. |
| 682 | |
| 683 | * "capabilities" <STRING> |
| 684 | |
| 685 | This a comma-separated list of capabilities supported by agent. Spaces must |
| 686 | be ignored, if any. |
| 687 | |
| 688 | At this time, if everything is ok for HAProxy (supported version and valid |
| 689 | max-frame-size value), the HELLO handshake is successfully completed. Else, |
| 690 | HAProxy sends a HAPROXY-DISCONNECT frame with the corresponding error. |
| 691 | |
Christopher Faulet | ba7bc16 | 2016-11-07 21:07:38 +0100 | [diff] [blame] | 692 | If "healthcheck" item was set to TRUE in the HAPROXY-HELLO frame, the agent can |
| 693 | safely close the connection without DISCONNECT frame. In all cases, HAProxy |
| 694 | will close the connexion at the end of the health check. |
| 695 | |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 696 | 3.2.6. Frame: NOTIFY |
| 697 | --------------------- |
| 698 | |
| 699 | Information are sent to the agents inside NOTIFY frames. These frames are |
| 700 | attached to a stream, so STREAM-ID and FRAME-ID must be set. The payload of |
| 701 | NOTIFY frames is a LIST-OF-MESSAGES and, if supported by agents, it can be |
| 702 | fragmented. |
| 703 | |
| 704 | NOTIFY frames must be acknowledge by agents sending an ACK frame, repeating |
| 705 | right STREAM-ID and FRAME-ID. |
| 706 | |
| 707 | 3.2.7. Frame: ACK |
| 708 | ------------------ |
| 709 | |
| 710 | ACK frames must be sent by agents to reply to NOTIFY frames. STREAM-ID and |
| 711 | FRAME-ID found in a NOTIFY frame must be reuse in the corresponding ACK |
| 712 | frame. The payload of ACK frames is a LIST-OF-ACTIONS and, if supported by |
| 713 | HAProxy, it can be fragmented. |
| 714 | |
| 715 | 3.2.8. Frame: HAPROXY-DISCONNECT |
| 716 | --------------------------------- |
| 717 | |
| 718 | If an error occurs, at anytime, from the HAProxy side, a HAPROXY-DISCONNECT |
| 719 | frame is sent with information describing the error. HAProxy will wait an |
| 720 | AGENT-DISCONNECT frame in reply. All other frames will be ignored. The agent |
| 721 | must then close the socket. |
| 722 | |
| 723 | The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and |
| 724 | FRAME-ID are must be set 0. |
| 725 | |
| 726 | Following items are mandatory in the KV-LIST: |
| 727 | |
| 728 | * "status-code" <UINT32> |
| 729 | |
| 730 | This is the code corresponding to the error. |
| 731 | |
| 732 | * "message" <STRING> |
| 733 | |
| 734 | This is a textual message describing the error. |
| 735 | |
| 736 | For more information about known errors, see section "Errors & timeouts" |
| 737 | |
| 738 | 3.2.9. Frame: AGENT-DISCONNECT |
| 739 | ------------------------------- |
| 740 | |
| 741 | If an error occurs, at anytime, from the agent size, a AGENT-DISCONNECT frame |
| 742 | is sent, with information desribing the error. such frame is also sent in reply |
| 743 | to a HAPROXY-DISCONNECT. The agent must close the socket just after sending |
| 744 | this frame. |
| 745 | |
| 746 | The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and |
| 747 | FRAME-ID are must be set 0. |
| 748 | |
| 749 | Following items are mandatory in the KV-LIST: |
| 750 | |
| 751 | * "status-code" <UINT32> |
| 752 | |
| 753 | This is the code corresponding to the error. |
| 754 | |
| 755 | * "message" <STRING> |
| 756 | |
| 757 | This is a textual message describing the error. |
| 758 | |
| 759 | For more information about known errors, see section "Errors & timeouts" |
| 760 | |
| 761 | 3.3. Events & Messages |
| 762 | ----------------------- |
| 763 | |
| 764 | Information about streams are sent in NOTIFY frames. You can specify which kind |
| 765 | of information to send by defining "spoe-message" sections in your SPOE |
| 766 | configuration file. for each "spoe-message" there will be a message in a NOTIFY |
| 767 | frame when the right event is triggered. |
| 768 | |
| 769 | A NOTIFY frame is sent for an specific event when there is at least one |
| 770 | "spoe-message" attached to this event. All messages for an event will be added |
| 771 | in the same NOTIFY frame. |
| 772 | |
| 773 | Here is the list of supported events: |
| 774 | |
| 775 | * on-client-session is triggered when a new client session is created. |
| 776 | This event is only available for SPOE filters |
| 777 | declared in a frontend or a listen section. |
| 778 | |
| 779 | * on-frontend-tcp-request is triggered just before the evaluation of |
| 780 | "tcp-request content" rules on the frontend side. |
| 781 | This event is only available for SPOE filters |
| 782 | declared in a frontend or a listen section. |
| 783 | |
| 784 | * on-backend-tcp-request is triggered just before the evaluation of |
| 785 | "tcp-request content" rules on the backend side. |
| 786 | This event is skipped for SPOE filters declared |
| 787 | in a listen section. |
| 788 | |
| 789 | * on-frontend-http-request is triggered just before the evaluation of |
| 790 | "http-request" rules on the frontend side. This |
| 791 | event is only available for SPOE filters declared |
| 792 | in a frontend or a listen section. |
| 793 | |
| 794 | * on-backend-http-request is triggered just before the evaluation of |
| 795 | "http-request" rules on the backend side. This |
| 796 | event is skipped for SPOE filters declared in a |
| 797 | listen section. |
| 798 | |
| 799 | * on-server-session is triggered when the session with the server is |
| 800 | established. |
| 801 | |
| 802 | * on-tcp-response is triggered just before the evaluation of |
| 803 | "tcp-response content" rules. |
| 804 | |
| 805 | * on-http-response is triggered just before the evaluation of |
| 806 | "http-response" rules. |
| 807 | |
| 808 | |
| 809 | The stream processing will loop on these events, when triggered, waiting the |
| 810 | agent reply. |
| 811 | |
| 812 | 3.4. Actions |
| 813 | ------------- |
| 814 | |
| 815 | An agent must acknowledge each NOTIFY frame by sending the corresponding ACK |
| 816 | frame. Actions can be added in these frames to dynamically take action on the |
| 817 | processing of a stream. |
| 818 | |
| 819 | Here is the list of supported actions: |
| 820 | |
| 821 | * set-var set the value for an existing variable. 3 arguments must be |
| 822 | attached to this action: the variable scope (proc, sess, txn, |
| 823 | req or req), the variable name (a string) and its value. |
| 824 | |
| 825 | ACTION-SET-VAR : <SET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME><VAR-VALUE> |
| 826 | |
| 827 | SET-VAR : <1> |
| 828 | NB-ARGS : <3> |
| 829 | VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE> |
| 830 | VAR-NAME : <STRING> |
| 831 | VAR-VALUE : <TYPED-DATA> |
| 832 | |
| 833 | PROCESS : <0> |
| 834 | SESSION : <1> |
| 835 | TRANSACTION : <2> |
| 836 | REQUEST : <3> |
| 837 | RESERVED : <4> |
| 838 | |
| 839 | * unset-var unset the value for an existing variable. 2 arguments must be |
| 840 | attached to this action: the variable scope (proc, sess, txn, |
| 841 | req or req) and the variable name (a string). |
| 842 | |
Christopher Faulet | 1002aac | 2016-12-09 17:41:54 +0100 | [diff] [blame] | 843 | ACTION-UNSET-VAR : <UNSET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME> |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 844 | |
Christopher Faulet | 1002aac | 2016-12-09 17:41:54 +0100 | [diff] [blame] | 845 | UNSET-VAR : <2> |
| 846 | NB-ARGS : <2> |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 847 | VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE> |
| 848 | VAR-NAME : <STRING> |
| 849 | |
| 850 | PROCESS : <0> |
| 851 | SESSION : <1> |
| 852 | TRANSACTION : <2> |
| 853 | REQUEST : <3> |
| 854 | RESERVED : <4> |
| 855 | |
| 856 | |
| 857 | NOTE: Name of the variables will be automatically prefixed by HAProxy to avoid |
| 858 | name clashes with other variables used in HAProxy. Moreover, unknown |
| 859 | variable will be silently ignored. |
| 860 | |
| 861 | 3.5. Error & timeouts |
| 862 | ---------------------- |
| 863 | |
| 864 | Here is the list of all known errors: |
| 865 | |
| 866 | STATUS CODE | DESCRIPTION |
| 867 | ----------------+-------------------------------------------------------- |
| 868 | 0 | normal (no error occurred) |
| 869 | 1 | I/O error |
| 870 | 2 | A timeout occurred |
| 871 | 3 | frame is too big |
| 872 | 4 | invalid frame received |
| 873 | 5 | version value not found |
| 874 | 6 | max-frame-size value not found |
| 875 | 7 | capabilities value not found |
| 876 | 8 | unsupported version |
| 877 | 9 | max-frame-size too big or too small |
| 878 | 99 | an unknown error occurrde |
| 879 | ----------------+-------------------------------------------------------- |
| 880 | |
| 881 | An agent can define its own errors using a not yet assigned status code. |
| 882 | |
Christopher Faulet | ea62c2a | 2016-11-14 10:54:21 +0100 | [diff] [blame] | 883 | IMPORTANT NOTE: By default, for a specific stream, when an abnormal/unexpected |
| 884 | error occurs, the SPOE is disabled for all the transaction. So |
| 885 | if you have several events configured, such error on an event |
| 886 | will disabled all followings. For TCP streams, this will |
| 887 | disable the SPOE for the whole session. For HTTP streams, this |
| 888 | will disable it for the transaction (request and response). |
| 889 | See 'option continue-on-error' to bypass this limitation. |
Christopher Faulet | f7e4e7e | 2016-10-27 22:29:49 +0200 | [diff] [blame] | 890 | |
| 891 | To avoid a stream to wait infinitly, you must carefully choose the |
| 892 | acknowledgement timeout. In most of cases, it will be quiet low. But it depends |
| 893 | on the responsivness of your service. |
| 894 | |
| 895 | You must also choose idle timeout carefully. Because connection with your |
| 896 | service depends on the backend configuration used by the SPOA, it is important |
| 897 | to use a lower value for idle timeout than the server timeout. Else the |
| 898 | connection will be closed by HAProxy. The same is true for hello timeout. You |
| 899 | should choose a lower value than the connect timeout. |
| 900 | |
| 901 | |
| 902 | /* |
| 903 | * Local variables: |
| 904 | * fill-column: 79 |
| 905 | * End: |
| 906 | */ |