Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 1 | 2015/05/02 Willy Tarreau |
Willy Tarreau | a339395 | 2014-05-10 15:16:43 +0200 | [diff] [blame] | 2 | HAProxy Technologies |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 3 | The PROXY protocol |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 4 | Versions 1 & 2 |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 5 | |
| 6 | Abstract |
| 7 | |
| 8 | The PROXY protocol provides a convenient way to safely transport connection |
| 9 | information such as a client's address across multiple layers of NAT or TCP |
| 10 | proxies. It is designed to require little changes to existing components and |
| 11 | to limit the performance impact caused by the processing of the transported |
| 12 | information. |
| 13 | |
| 14 | |
| 15 | Revision history |
| 16 | |
| 17 | 2010/10/29 - first version |
| 18 | 2011/03/20 - update: implementation and security considerations |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 19 | 2012/06/21 - add support for binary format |
| 20 | 2012/11/19 - final review and fixes |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 21 | 2014/05/18 - modify and extend PROXY protocol version 2 |
Willy Tarreau | 7a6f134 | 2014-06-14 11:45:09 +0200 | [diff] [blame] | 22 | 2014/06/11 - fix example code to consider ver+cmd merge |
| 23 | 2014/06/14 - fix v2 header check in example code, and update Forwarded spec |
Willy Tarreau | 87b7ac9 | 2014-07-25 08:56:07 +0200 | [diff] [blame] | 24 | 2014/07/12 - update list of implementations (add Squid) |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 25 | 2015/05/02 - update list of implementations and format of the TLV add-ons |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 26 | |
| 27 | |
| 28 | 1. Background |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 29 | |
| 30 | Relaying TCP connections through proxies generally involves a loss of the |
| 31 | original TCP connection parameters such as source and destination addresses, |
| 32 | ports, and so on. Some protocols make it a little bit easier to transfer such |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 33 | information. For SMTP, Postfix authors have proposed the XCLIENT protocol [1] |
Willy Tarreau | 7a6f134 | 2014-06-14 11:45:09 +0200 | [diff] [blame] | 34 | which received broad adoption and is particularly suited to mail exchanges. |
| 35 | For HTTP, there is the "Forwarded" extension [2], which aims at replacing the |
| 36 | omnipresent "X-Forwarded-For" header which carries information about the |
| 37 | original source address, and the less common X-Original-To which carries |
| 38 | information about the destination address. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 39 | |
| 40 | However, both mechanisms require a knowledge of the underlying protocol to be |
| 41 | implemented in intermediaries. |
| 42 | |
| 43 | Then comes a new class of products which we'll call "dumb proxies", not because |
| 44 | they don't do anything, but because they're processing protocol-agnostic data. |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 45 | Both Stunnel[3] and Stud[4] are examples of such "dumb proxies". They talk raw |
| 46 | TCP on one side, and raw SSL on the other one, and do that reliably, without |
Willy Tarreau | 7a6f134 | 2014-06-14 11:45:09 +0200 | [diff] [blame] | 47 | any knowledge of what protocol is transported on top of the connection. Haproxy |
| 48 | running in pure TCP mode obviously falls into that category as well. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 49 | |
| 50 | The problem with such a proxy when it is combined with another one such as |
Willy Tarreau | 7a6f134 | 2014-06-14 11:45:09 +0200 | [diff] [blame] | 51 | haproxy, is to adapt it to talk the higher level protocol. A patch is available |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 52 | for Stunnel to make it capable of inserting an X-Forwarded-For header in the |
| 53 | first HTTP request of each incoming connection. Haproxy is able not to add |
| 54 | another one when the connection comes from Stunnel, so that it's possible to |
| 55 | hide it from the servers. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 56 | |
| 57 | The typical architecture becomes the following one : |
| 58 | |
| 59 | |
| 60 | +--------+ HTTP :80 +----------+ |
| 61 | | client | --------------------------------> | | |
| 62 | | | | haproxy, | |
| 63 | +--------+ +---------+ | 1 or 2 | |
| 64 | / / HTTPS | stunnel | HTTP :81 | listening| |
| 65 | <________/ ---------> | (server | ---------> | ports | |
| 66 | | mode) | | | |
| 67 | +---------+ +----------+ |
| 68 | |
| 69 | |
| 70 | The problem appears when haproxy runs with keep-alive on the side towards the |
| 71 | client. The Stunnel patch will only add the X-Forwarded-For header to the first |
| 72 | request of each connection and all subsequent requests will not have it. One |
| 73 | solution could be to improve the patch to make it support keep-alive and parse |
| 74 | all forwarded data, whether they're announced with a Content-Length or with a |
| 75 | Transfer-Encoding, taking care of special methods such as HEAD which announce |
| 76 | data without transfering them, etc... In fact, it would require implementing a |
| 77 | full HTTP stack in Stunnel. It would then become a lot more complex, a lot less |
| 78 | reliable and would not anymore be the "dumb proxy" that fits every purposes. |
| 79 | |
| 80 | In practice, we don't need to add a header for each request because we'll emit |
| 81 | the exact same information every time : the information related to the client |
| 82 | side connection. We could then cache that information in haproxy and use it for |
| 83 | every other request. But that becomes dangerous and is still limited to HTTP |
| 84 | only. |
| 85 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 86 | Another approach consists in prepending each connection with a header reporting |
| 87 | the characteristics of the other side's connection. This method is simpler to |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 88 | implement, does not require any protocol-specific knowledge on either side, and |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 89 | completely fits the purpose since what is desired precisely is to know the |
| 90 | other side's connection endpoints. It is easy to perform for the sender (just |
| 91 | send a short header once the connection is established) and to parse for the |
| 92 | receiver (simply perform one read() on the incoming connection to fill in |
| 93 | addresses after an accept). The protocol used to carry connection information |
| 94 | across proxies was thus called the PROXY protocol. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 95 | |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 96 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 97 | 2. The PROXY protocol header |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 98 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 99 | This document uses a few terms that are worth explaining here : |
| 100 | - "connection initiator" is the party requesting a new connection |
| 101 | - "connection target" is the party accepting a connection request |
| 102 | - "client" is the party for which a connection was requested |
| 103 | - "server" is the party to which the client desired to connect |
| 104 | - "proxy" is the party intercepting and relaying the connection |
| 105 | from the client to the server. |
| 106 | - "sender" is the party sending data over a connection. |
| 107 | - "receiver" is the party receiving data from the sender. |
| 108 | - "header" or "PROXY protocol header" is the block of connection information |
| 109 | the connection initiator prepends at the beginning of a connection, which |
| 110 | makes it the sender from the protocol point of view. |
| 111 | |
| 112 | The PROXY protocol's goal is to fill the server's internal structures with the |
| 113 | information collected by the proxy that the server would have been able to get |
| 114 | by itself if the client was connecting directly to the server instead of via a |
| 115 | proxy. The information carried by the protocol are the ones the server would |
| 116 | get using getsockname() and getpeername() : |
| 117 | - address family (AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX) |
| 118 | - socket protocol (SOCK_STREAM for TCP, SOCK_DGRAM for UDP) |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 119 | - layer 3 source and destination addresses |
| 120 | - layer 4 source and destination ports if any |
| 121 | |
| 122 | Unlike the XCLIENT protocol, the PROXY protocol was designed with limited |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 123 | extensibility in order to help the receiver parse it very fast. Version 1 was |
| 124 | focused on keeping it human-readable for better debugging possibilities, which |
| 125 | is always desirable for early adoption when few implementations exist. Version |
| 126 | 2 adds support for a binary encoding of the header which is much more efficient |
| 127 | to produce and to parse, especially when dealing with IPv6 addresses that are |
| 128 | expensive to emit in ASCII form and to parse. |
| 129 | |
| 130 | In both cases, the protocol simply consists in an easily parsable header placed |
| 131 | by the connection initiator at the beginning of each connection. The protocol |
| 132 | is intentionally stateless in that it does not expect the sender to wait for |
| 133 | the receiver before sending the header, nor the receiver to send anything back. |
| 134 | |
| 135 | This specification supports two header formats, a human-readable format which |
| 136 | is the only format supported in version 1 of the protocol, and a binary format |
| 137 | which is only supported in version 2. Both formats were designed to ensure that |
| 138 | the header cannot be confused with common higher level protocols such as HTTP, |
| 139 | SSL/TLS, FTP or SMTP, and that both formats are easily distinguishable one from |
| 140 | each other for the receiver. |
| 141 | |
| 142 | Version 1 senders MAY only produce the human-readable header format. Version 2 |
| 143 | senders MAY only produce the binary header format. Version 1 receivers MUST at |
| 144 | least implement the human-readable header format. Version 2 receivers MUST at |
| 145 | least implement the binary header format, and it is recommended that they also |
| 146 | implement the human-readable header format for better interoperability and ease |
| 147 | of upgrade when facing version 1 senders. |
| 148 | |
| 149 | Both formats are designed to fit in the smallest TCP segment that any TCP/IP |
| 150 | host is required to support (576 - 40 = 536 bytes). This ensures that the whole |
| 151 | header will always be delivered at once when the socket buffers are still empty |
| 152 | at the beginning of a connection. The sender must always ensure that the header |
| 153 | is sent at once, so that the transport layer maintains atomicity along the path |
| 154 | to the receiver. The receiver may be tolerant to partial headers or may simply |
| 155 | drop the connection when receiving a partial header. Recommendation is to be |
| 156 | tolerant, but implementation constraints may not always easily permit this. It |
| 157 | is important to note that nothing forces any intermediary to forward the whole |
| 158 | header at once, because TCP is a streaming protocol which may be processed one |
| 159 | byte at a time if desired, causing the header to be fragmented when reaching |
| 160 | the receiver. But due to the places where such a protocol is used, the above |
| 161 | simplification generally is acceptable because the risk of crossing such a |
| 162 | device handling one byte at a time is close to zero. |
| 163 | |
| 164 | The receiver MUST NOT start processing the connection before it receives a |
| 165 | complete and valid PROXY protocol header. This is particularly important for |
| 166 | protocols where the receiver is expected to speak first (eg: SMTP, FTP or SSH). |
| 167 | The receiver may apply a short timeout and decide to abort the connection if |
| 168 | the protocol header is not seen within a few seconds (at least 3 seconds to |
| 169 | cover a TCP retransmit). |
| 170 | |
| 171 | The receiver MUST be configured to only receive the protocol described in this |
| 172 | specification and MUST not try to guess whether the protocol header is present |
| 173 | or not. This means that the protocol explicitly prevents port sharing between |
| 174 | public and private access. Otherwise it would open a major security breach by |
| 175 | allowing untrusted parties to spoof their connection addresses. The receiver |
| 176 | SHOULD ensure proper access filtering so that only trusted proxies are allowed |
| 177 | to use this protocol. |
| 178 | |
| 179 | Some proxies are smart enough to understand transported protocols and to reuse |
| 180 | idle server connections for multiple messages. This typically happens in HTTP |
| 181 | where requests from multiple clients may be sent over the same connection. Such |
| 182 | proxies MUST NOT implement this protocol on multiplexed connections because the |
| 183 | receiver would use the address advertised in the PROXY header as the address of |
| 184 | all forwarded requests's senders. In fact, such proxies are not dumb proxies, |
| 185 | and since they do have a complete understanding of the transported protocol, |
| 186 | they MUST use the facilities provided by this protocol to present the client's |
| 187 | address. |
| 188 | |
| 189 | |
| 190 | 2.1. Human-readable header format (Version 1) |
| 191 | |
| 192 | This is the format specified in version 1 of the protocol. It consists in one |
| 193 | line of ASCII text matching exactly the following block, sent immediately and |
| 194 | at once upon the connection establishment and prepended before any data flowing |
| 195 | from the sender to the receiver : |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 196 | |
| 197 | - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 ) |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 198 | Seeing this string indicates that this is version 1 of the protocol. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 199 | |
| 200 | - exactly one space : " " ( \x20 ) |
| 201 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 202 | - a string indicating the proxied INET protocol and family. As of version 1, |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 203 | only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6" |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 204 | ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Other, unsupported, |
| 205 | or unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E |
| 206 | \x4B \x4E \x4F \x57 \x4E ). For "UNKNOWN", the rest of the line before the |
| 207 | CRLF may be omitted by the sender, and the receiver must ignore anything |
| 208 | presented before the CRLF is found. Note that an earlier version of this |
| 209 | specification suggested to use this when sending health checks, but this |
| 210 | causes issues with servers that reject the "UNKNOWN" keyword. Thus is it |
| 211 | now recommended not to send "UNKNOWN" when the connection is expected to |
| 212 | be accepted, but only when it is not possible to correctly fill the PROXY |
| 213 | line. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 214 | |
| 215 | - exactly one space : " " ( \x20 ) |
| 216 | |
| 217 | - the layer 3 source address in its canonical format. IPv4 addresses must be |
| 218 | indicated as a series of exactly 4 integers in the range [0..255] inclusive |
| 219 | written in decimal representation separated by exactly one dot between each |
| 220 | other. Heading zeroes are not permitted in front of numbers in order to |
| 221 | avoid any possible confusion with octal numbers. IPv6 addresses must be |
| 222 | indicated as series of 4 hexadecimal digits (upper or lower case) delimited |
| 223 | by colons between each other, with the acceptance of one double colon |
| 224 | sequence to replace the largest acceptable range of consecutive zeroes. The |
| 225 | total number of decoded bits must exactly be 128. The advertised protocol |
| 226 | family dictates what format to use. |
| 227 | |
| 228 | - exactly one space : " " ( \x20 ) |
| 229 | |
| 230 | - the layer 3 destination address in its canonical format. It is the same |
| 231 | format as the layer 3 source address and matches the same family. |
| 232 | |
| 233 | - exactly one space : " " ( \x20 ) |
| 234 | |
| 235 | - the TCP source port represented as a decimal integer in the range |
| 236 | [0..65535] inclusive. Heading zeroes are not permitted in front of numbers |
| 237 | in order to avoid any possible confusion with octal numbers. |
| 238 | |
| 239 | - exactly one space : " " ( \x20 ) |
| 240 | |
| 241 | - the TCP destination port represented as a decimal integer in the range |
| 242 | [0..65535] inclusive. Heading zeroes are not permitted in front of numbers |
| 243 | in order to avoid any possible confusion with octal numbers. |
| 244 | |
| 245 | - the CRLF sequence ( \x0D \x0A ) |
| 246 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 247 | |
| 248 | The maximum line lengths the receiver must support including the CRLF are : |
| 249 | - TCP/IPv4 : |
| 250 | "PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n" |
| 251 | => 5 + 1 + 4 + 1 + 15 + 1 + 15 + 1 + 5 + 1 + 5 + 2 = 56 chars |
| 252 | |
| 253 | - TCP/IPv6 : |
| 254 | "PROXY TCP6 ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n" |
| 255 | => 5 + 1 + 4 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 104 chars |
| 256 | |
| 257 | - unknown connection (short form) : |
| 258 | "PROXY UNKNOWN\r\n" |
| 259 | => 5 + 1 + 7 + 2 = 15 chars |
| 260 | |
| 261 | - worst case (optional fields set to 0xff) : |
| 262 | "PROXY UNKNOWN ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n" |
| 263 | => 5 + 1 + 7 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 107 chars |
| 264 | |
| 265 | So a 108-byte buffer is always enough to store all the line and a trailing zero |
| 266 | for string processing. |
| 267 | |
| 268 | The receiver must wait for the CRLF sequence before starting to decode the |
| 269 | addresses in order to ensure they are complete and properly parsed. If the CRLF |
| 270 | sequence is not found in the first 107 characters, the receiver should declare |
| 271 | the line invalid. A receiver may reject an incomplete line which does not |
| 272 | contain the CRLF sequence in the first atomic read operation. The receiver must |
| 273 | not tolerate a single CR or LF character to end the line when a complete CRLF |
| 274 | sequence is expected. |
| 275 | |
| 276 | Any sequence which does not exactly match the protocol must be discarded and |
| 277 | cause the receiver to abort the connection. It is recommended to abort the |
| 278 | connection as soon as possible so that the sender gets a chance to notice the |
| 279 | anomaly and log it. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 280 | |
| 281 | If the announced transport protocol is "UNKNOWN", then the receiver knows that |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 282 | the sender speaks the correct PROXY protocol with the appropriate version, and |
| 283 | SHOULD accept the connection and use the real connection's parameters as if |
| 284 | there were no PROXY protocol header on the wire. However, senders SHOULD not |
| 285 | use the "UNKNOWN" protocol when they are the initiators of outgoing connections |
| 286 | because some receivers may reject them. When a load balancing proxy has to send |
| 287 | health checks to a server, it SHOULD build a valid PROXY line which it will |
| 288 | fill with a getsockname()/getpeername() pair indicating the addresses used. It |
| 289 | is important to understand that doing so is not appropriate when some source |
| 290 | address translation is performed between the sender and the receiver. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 291 | |
| 292 | An example of such a line before an HTTP request would look like this (CR |
| 293 | marked as "\r" and LF marked as "\n") : |
| 294 | |
| 295 | PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n |
| 296 | GET / HTTP/1.1\r\n |
| 297 | Host: 192.168.0.11\r\n |
| 298 | \r\n |
| 299 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 300 | For the sender, the header line is easy to put into the output buffers once the |
| 301 | connection is established. Note that since the line is always shorter than an |
| 302 | MSS, the sender is guaranteed to always be able to emit it at once and should |
| 303 | not even bother handling partial sends. For the receiver, once the header is |
| 304 | parsed, it is easy to skip it from the input buffers. Please consult section 9 |
| 305 | for implementation suggestions. |
| 306 | |
| 307 | |
| 308 | 2.2. Binary header format (version 2) |
| 309 | |
| 310 | Producing human-readable IPv6 addresses and parsing them is very inefficient, |
| 311 | due to the multiple possible representation formats and the handling of compact |
| 312 | address format. It was also not possible to specify address families outside |
| 313 | IPv4/IPv6 nor non-TCP protocols. Another drawback of the human-readable format |
| 314 | is the fact that implementations need to parse all characters to find the |
| 315 | trailing CRLF, which makes it harder to read only the exact bytes count. Last, |
| 316 | the UNKNOWN address type has not always been accepted by servers as a valid |
| 317 | protocol because of its imprecise meaning. |
| 318 | |
| 319 | Version 2 of the protocol thus introduces a new binary format which remains |
| 320 | distinguishable from version 1 and from other commonly used protocols. It was |
| 321 | specially designed in order to be incompatible with a wide range of protocols |
| 322 | and to be rejected by a number of common implementations of these protocols |
| 323 | when unexpectedly presented (please see section 7). Also for better processing |
| 324 | efficiency, IPv4 and IPv6 addresses are respectively aligned on 4 and 16 bytes |
| 325 | boundaries. |
| 326 | |
| 327 | The binary header format starts with a constant 12 bytes block containing the |
| 328 | protocol signature : |
| 329 | |
| 330 | \x0D \x0A \x0D \x0A \x00 \x0D \x0A \x51 \x55 \x49 \x54 \x0A |
| 331 | |
| 332 | Note that this block contains a null byte at the 5th position, so it must not |
| 333 | be handled as a null-terminated string. |
| 334 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 335 | The next byte (the 13th one) is the protocol version and command. |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 336 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 337 | The highest four bits contains the version. As of this specification, it must |
| 338 | always be sent as \x2 and the receiver must only accept this value. |
| 339 | |
| 340 | The lowest four bits represents the command : |
| 341 | - \x0 : LOCAL : the connection was established on purpose by the proxy |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 342 | without being relayed. The connection endpoints are the sender and the |
| 343 | receiver. Such connections exist when the proxy sends health-checks to the |
| 344 | server. The receiver must accept this connection as valid and must use the |
| 345 | real connection endpoints and discard the protocol block including the |
| 346 | family which is ignored. |
| 347 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 348 | - \x1 : PROXY : the connection was established on behalf of another node, |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 349 | and reflects the original connection endpoints. The receiver must then use |
| 350 | the information provided in the protocol block to get original the address. |
| 351 | |
| 352 | - other values are unassigned and must not be emitted by senders. Receivers |
| 353 | must drop connections presenting unexpected values here. |
| 354 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 355 | The 14th byte contains the transport protocol and address family. The highest 4 |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 356 | bits contain the address family, the lowest 4 bits contain the protocol. |
| 357 | |
| 358 | The address family maps to the original socket family without necessarily |
| 359 | matching the values internally used by the system. It may be one of : |
| 360 | |
| 361 | - 0x0 : AF_UNSPEC : the connection is forwarded for an unknown, unspecified |
| 362 | or unsupported protocol. The sender should use this family when sending |
| 363 | LOCAL commands or when dealing with unsupported protocol families. The |
| 364 | receiver is free to accept the connection anyway and use the real endpoint |
| 365 | addresses or to reject it. The receiver should ignore address information. |
| 366 | |
| 367 | - 0x1 : AF_INET : the forwarded connection uses the AF_INET address family |
| 368 | (IPv4). The addresses are exactly 4 bytes each in network byte order, |
| 369 | followed by transport protocol information (typically ports). |
| 370 | |
| 371 | - 0x2 : AF_INET6 : the forwarded connection uses the AF_INET6 address family |
| 372 | (IPv6). The addresses are exactly 16 bytes each in network byte order, |
| 373 | followed by transport protocol information (typically ports). |
| 374 | |
| 375 | - 0x3 : AF_UNIX : the forwarded connection uses the AF_UNIX address family |
| 376 | (UNIX). The addresses are exactly 108 bytes each. |
| 377 | |
| 378 | - other values are unspecified and must not be emitted in version 2 of this |
| 379 | protocol and must be rejected as invalid by receivers. |
| 380 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 381 | The transport protocol is specified in the lowest 4 bits of the the 14th byte : |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 382 | |
| 383 | - 0x0 : UNSPEC : the connection is forwarded for an unknown, unspecified |
| 384 | or unsupported protocol. The sender should use this family when sending |
| 385 | LOCAL commands or when dealing with unsupported protocol families. The |
| 386 | receiver is free to accept the connection anyway and use the real endpoint |
| 387 | addresses or to reject it. The receiver should ignore address information. |
| 388 | |
| 389 | - 0x1 : STREAM : the forwarded connection uses a SOCK_STREAM protocol (eg: |
| 390 | TCP or UNIX_STREAM). When used with AF_INET/AF_INET6 (TCP), the addresses |
| 391 | are followed by the source and destination ports represented on 2 bytes |
| 392 | each in network byte order. |
| 393 | |
| 394 | - 0x2 : DGRAM : the forwarded connection uses a SOCK_DGRAM protocol (eg: |
| 395 | UDP or UNIX_DGRAM). When used with AF_INET/AF_INET6 (UDP), the addresses |
| 396 | are followed by the source and destination ports represented on 2 bytes |
| 397 | each in network byte order. |
| 398 | |
| 399 | - other values are unspecified and must not be emitted in version 2 of this |
| 400 | protocol and must be rejected as invalid by receivers. |
| 401 | |
| 402 | In practice, the following protocol bytes are expected : |
| 403 | |
| 404 | - \x00 : UNSPEC : the connection is forwarded for an unknown, unspecified |
| 405 | or unsupported protocol. The sender should use this family when sending |
| 406 | LOCAL commands or when dealing with unsupported protocol families. When |
| 407 | used with a LOCAL command, the receiver must accept the connection and |
| 408 | ignore any address information. For other commands, the receiver is free |
| 409 | to accept the connection anyway and use the real endpoints addresses or to |
| 410 | reject the connection. The receiver should ignore address information. |
| 411 | |
| 412 | - \x11 : TCP over IPv4 : the forwarded connection uses TCP over the AF_INET |
| 413 | protocol family. Address length is 2*4 + 2*2 = 12 bytes. |
| 414 | |
| 415 | - \x12 : UDP over IPv4 : the forwarded connection uses UDP over the AF_INET |
| 416 | protocol family. Address length is 2*4 + 2*2 = 12 bytes. |
| 417 | |
| 418 | - \x21 : TCP over IPv6 : the forwarded connection uses TCP over the AF_INET6 |
| 419 | protocol family. Address length is 2*16 + 2*2 = 36 bytes. |
| 420 | |
| 421 | - \x22 : UDP over IPv6 : the forwarded connection uses UDP over the AF_INET6 |
| 422 | protocol family. Address length is 2*16 + 2*2 = 36 bytes. |
| 423 | |
| 424 | - \x31 : UNIX stream : the forwarded connection uses SOCK_STREAM over the |
| 425 | AF_UNIX protocol family. Address length is 2*108 = 216 bytes. |
| 426 | |
| 427 | - \x32 : UNIX datagram : the forwarded connection uses SOCK_DGRAM over the |
| 428 | AF_UNIX protocol family. Address length is 2*108 = 216 bytes. |
| 429 | |
| 430 | |
| 431 | Only the UNSPEC protocol byte (\x00) is mandatory. A receiver is not required |
| 432 | to implement other ones, provided that it automatically falls back to the |
| 433 | UNSPEC mode for the valid combinations above that it does not support. |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 434 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 435 | The 15th and 16th bytes is the address length in bytes in network endien order. |
| 436 | It is used so that the receiver knows how many address bytes to skip even when |
| 437 | it does not implement the presented protocol. Thus the length of the protocol |
| 438 | header in bytes is always exactly 16 + this value. When a sender presents a |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 439 | LOCAL connection, it should not present any address so it sets this field to |
| 440 | zero. Receivers MUST always consider this field to skip the appropriate number |
| 441 | of bytes and must not assume zero is presented for LOCAL connections. When a |
| 442 | receiver accepts an incoming connection showing an UNSPEC address family or |
| 443 | protocol, it may or may not decide to log the address information if present. |
| 444 | |
| 445 | So the 16-byte version 2 header can be described this way : |
| 446 | |
| 447 | struct proxy_hdr_v2 { |
| 448 | uint8_t sig[12]; /* hex 0D 0A 0D 0A 00 0D 0A 51 55 49 54 0A */ |
Willy Tarreau | 0f6093a | 2014-06-11 21:21:26 +0200 | [diff] [blame] | 449 | uint8_t ver_cmd; /* protocol version and command */ |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 450 | uint8_t fam; /* protocol family and address */ |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 451 | uint16_t len; /* number of following bytes part of the header */ |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 452 | }; |
| 453 | |
| 454 | Starting from the 17th byte, addresses are presented in network byte order. |
| 455 | The address order is always the same : |
| 456 | - source layer 3 address in network byte order |
| 457 | - destination layer 3 address in network byte order |
| 458 | - source layer 4 address if any, in network byte order (port) |
| 459 | - destination layer 4 address if any, in network byte order (port) |
| 460 | |
| 461 | The address block may directly be sent from or received into the following |
| 462 | union which makes it easy to cast from/to the relevant socket native structs |
| 463 | depending on the address type : |
| 464 | |
| 465 | union proxy_addr { |
| 466 | struct { /* for TCP/UDP over IPv4, len = 12 */ |
| 467 | uint32_t src_addr; |
| 468 | uint32_t dst_addr; |
| 469 | uint16_t src_port; |
| 470 | uint16_t dst_port; |
| 471 | } ipv4_addr; |
| 472 | struct { /* for TCP/UDP over IPv6, len = 36 */ |
| 473 | uint8_t src_addr[16]; |
| 474 | uint8_t dst_addr[16]; |
| 475 | uint16_t src_port; |
| 476 | uint16_t dst_port; |
| 477 | } ipv6_addr; |
| 478 | struct { /* for AF_UNIX sockets, len = 216 */ |
| 479 | uint8_t src_addr[108]; |
| 480 | uint8_t dst_addr[108]; |
| 481 | } unix_addr; |
| 482 | }; |
| 483 | |
| 484 | The sender must ensure that all the protocol header is sent at once. This block |
| 485 | is always smaller than an MSS, so there is no reason for it to be segmented at |
| 486 | the beginning of the connection. The receiver should also process the header |
| 487 | at once. The receiver must not start to parse an address before the whole |
| 488 | address block is received. The receiver must also reject incoming connections |
| 489 | containing partial protocol headers. |
| 490 | |
| 491 | A receiver may be configured to support both version 1 and version 2 of the |
| 492 | protocol. Identifying the protocol version is easy : |
| 493 | |
| 494 | - if the incoming byte count is 16 or above and the 13 first bytes match |
| 495 | the protocol signature block followed by the protocol version 2 : |
| 496 | |
| 497 | \x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A\x02 |
| 498 | |
| 499 | - otherwise, if the incoming byte count is 8 or above, and the 5 first |
| 500 | characters match the ASCII representation of "PROXY" then the protocol |
| 501 | must be parsed as version 1 : |
| 502 | |
| 503 | \x50\x52\x4F\x58\x59 |
| 504 | |
| 505 | - otherwise the protocol is not covered by this specification and the |
| 506 | connection must be dropped. |
| 507 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 508 | If the length specified in the PROXY protocol header indicates that additional |
| 509 | bytes are part of the header beyond the address information, a receiver may |
| 510 | choose to skip over and ignore those bytes, or attempt to interpret those |
| 511 | bytes. |
| 512 | |
| 513 | The information in those bytes will be arranged in Type-Length-Value (TLV |
| 514 | vectors) in the following format. The first byte is the Type of the vector. |
| 515 | The second two bytes represent the length in bytes of the value (not included |
| 516 | the Type and Length bytes), and following the length field is the number of |
| 517 | bytes specified by the length. |
| 518 | |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 519 | struct pp2_tlv { |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 520 | uint8_t type; |
| 521 | uint8_t length_hi; |
| 522 | uint8_t length_lo; |
| 523 | uint8_t value[0]; |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 524 | }; |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 525 | |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 526 | The following types have already been registered for the <type> field : |
| 527 | |
| 528 | #define PP2_TYPE_ALPN 0x01 |
| 529 | #define PP2_TYPE_AUTHORITY 0x02 |
| 530 | #define PP2_TYPE_SSL 0x20 |
| 531 | #define PP2_TYPE_SSL_VERSION 0x21 |
| 532 | #define PP2_TYPE_SSL_CN 0x22 |
| 533 | #define PP2_TYPE_NETNS 0x30 |
| 534 | |
| 535 | For the type PP2_TYPE_SSL, the value is itselv a defined like this : |
| 536 | |
| 537 | struct pp2_tlv_ssl { |
| 538 | uint8_t client; |
| 539 | uint32_t verify; |
| 540 | struct pp2_tlv sub_tlv[0]; |
| 541 | }; |
| 542 | |
| 543 | And the <client> field is made of a bit field from the following values, |
| 544 | indicating which element is present : |
| 545 | |
| 546 | #define PP2_CLIENT_SSL 0x01 |
| 547 | #define PP2_CLIENT_CERT_CONN 0x02 |
| 548 | #define PP2_CLIENT_CERT_SESS 0x04 |
| 549 | |
| 550 | Each of these elements may lead to extra data being appended to this TLV using |
| 551 | a second level of TLV encapsulation. It is thus possible to find multiple TLV |
| 552 | values after this field. The total length of the upper TLV will reflect this. |
| 553 | |
| 554 | PP2_CLIENT_SSL indicates that the client connected over SSL/TLS. When this |
| 555 | field is present, the string representation of the TLS version is appended at |
| 556 | the end of the field in the TLV format using the type PP2_TYPE_SSL_VERSION. |
| 557 | |
| 558 | PP2_CLIENT_CERT_CONN indicates that the client provided a certificate over the |
| 559 | current connection. PP2_CLIENT_CERT_SESS indicates that the client provided a |
| 560 | certificate at least once over the TLS session this connection belongs to. In |
| 561 | both cases, the string representation of the client certificate's CN may be |
| 562 | appended after the SSL/TLS version using the TLV format using the type |
| 563 | PP2_TYPE_SSL_CN. |
| 564 | |
| 565 | The type PP2_TYPE_NETNS defines the value as the string representation of the |
| 566 | namespace's name. |
| 567 | |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 568 | |
| 569 | 3. Implementations |
| 570 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 571 | Haproxy 1.5 implements version 1 of the PROXY protocol on both sides : |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 572 | - the listening sockets accept the protocol when the "accept-proxy" setting |
| 573 | is passed to the "bind" keyword. Connections accepted on such listeners |
| 574 | will behave just as if the source really was the one advertised in the |
| 575 | protocol. This is true for logging, ACLs, content filtering, transparent |
| 576 | proxying, etc... |
| 577 | |
| 578 | - the protocol may be used to connect to servers if the "send-proxy" setting |
| 579 | is present on the "server" line. It is enabled on a per-server basis, so it |
| 580 | is possible to have it enabled for remote servers only and still have local |
| 581 | ones behave differently. If the incoming connection was accepted with the |
| 582 | "accept-proxy", then the relayed information is the one advertised in this |
| 583 | connection's PROXY line. |
| 584 | |
David S | afb7683 | 2014-05-08 23:42:08 -0400 | [diff] [blame] | 585 | - Haproxy 1.5 also implements version 2 of the PROXY protocol as a sender. In |
| 586 | addition, a TLV with limited, optional, SSL information has been added. |
| 587 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 588 | Stunnel added support for version 1 of the protocol for outgoing connections in |
| 589 | version 4.45. |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 590 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 591 | Stud added support for version 1 of the protocol for outgoing connections on |
| 592 | 2011/06/29. |
| 593 | |
| 594 | Postfix added support for version 1 of the protocol for incoming connections |
| 595 | in smtpd and postscreen in version 2.10. |
| 596 | |
| 597 | A patch is available for Stud[5] to implement version 1 of the protocol on |
| 598 | incoming connections. |
| 599 | |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 600 | Support for versions 1 and 2 of the protocol was added to Varnish 4.1 [6]. |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 601 | |
Todd Lyons | d1dcea0 | 2014-06-03 13:29:33 -0700 | [diff] [blame] | 602 | Exim added support for version 1 and version 2 of the protocol for incoming |
| 603 | connections on 2014/05/13, and will be released as part of version 4.83. |
| 604 | |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 605 | Squid added support for versions 1 and 2 of the protocol in version 3.5 [7]. |
| 606 | |
| 607 | Jetty 9.3.0 supports protocol version 1. |
| 608 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 609 | The protocol is simple enough that it is expected that other implementations |
| 610 | will appear, especially in environments such as SMTP, IMAP, FTP, RDP where the |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 611 | client's address is an important piece of information for the server and some |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 612 | intermediaries. In fact, several proprietary deployments have already done so |
| 613 | on FTP and SMTP servers. |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 614 | |
| 615 | Proxy developers are encouraged to implement this protocol, because it will |
| 616 | make their products much more transparent in complex infrastructures, and will |
| 617 | get rid of a number of issues related to logging and access control. |
| 618 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 619 | |
| 620 | 4. Architectural benefits |
| 621 | 4.1. Multiple layers |
| 622 | |
| 623 | Using the PROXY protocol instead of transparent proxy provides several benefits |
| 624 | in multiple-layer infrastructures. The first immediate benefit is that it |
| 625 | becomes possible to chain multiple layers of proxies and always present the |
| 626 | original IP address. for instance, let's consider the following 2-layer proxy |
| 627 | architecture : |
| 628 | |
| 629 | Internet |
| 630 | ,---. | client to PX1: |
| 631 | ( X ) | native protocol |
| 632 | `---' | |
| 633 | | V |
| 634 | +--+--+ +-----+ |
| 635 | | FW1 |------| PX1 | |
| 636 | +--+--+ +-----+ | PX1 to PX2: PROXY + native |
| 637 | | V |
| 638 | +--+--+ +-----+ |
| 639 | | FW2 |------| PX2 | |
| 640 | +--+--+ +-----+ | PX2 to SRV: PROXY + native |
| 641 | | V |
| 642 | +--+--+ |
| 643 | | SRV | |
| 644 | +-----+ |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 645 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 646 | Firewall FW1 receives traffic from internet-based clients and forwards it to |
| 647 | reverse-proxy PX1. PX1 adds a PROXY header then forwards to PX2 via FW2. PX2 |
| 648 | is configured to read the PROXY header and to emit it on output. It then joins |
| 649 | the origin server SRV and presents the original client's address there. Since |
| 650 | all TCP connections endpoints are real machines and are not spoofed, there is |
| 651 | no issue for the return traffic to pass via the firewalls and reverse proxies. |
| 652 | Using transparent proxy, this would be quite difficult because the firewalls |
| 653 | would have to deal with the client's address coming from the proxies in the DMZ |
| 654 | and would have to correctly route the return traffic there instead of using the |
| 655 | default route. |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 656 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 657 | |
| 658 | 4.2. IPv4 and IPv6 integration |
| 659 | |
| 660 | The protocol also eases IPv4 and IPv6 integration : if only the first layer |
| 661 | (FW1 and PX1) is IPv6-capable, it is still possible to present the original |
| 662 | client's IPv6 address to the target server eventhough the whole chain is only |
| 663 | connected via IPv4. |
| 664 | |
| 665 | |
| 666 | 4.3. Multiple return paths |
| 667 | |
| 668 | When transparent proxy is used, it is not possible to run multiple proxies |
| 669 | because the return traffic would follow the default route instead of finding |
| 670 | the proper proxy. Some tricks are sometimes possible using multiple server |
| 671 | addresses and policy routing but these are very limited. |
| 672 | |
| 673 | Using the PROXY protocol, this problem disappears as the servers don't need |
| 674 | to route to the client, just to the proxy that forwarded the connection. So |
| 675 | it is perfectly possible to run a proxy farm in front of a very large server |
| 676 | farm and have it working effortless, even when dealing with multiple sites. |
| 677 | |
| 678 | This is particularly important in Cloud-like environments where there is little |
| 679 | choice of binding to random addresses and where the lower processing power per |
| 680 | node generally requires multiple front nodes. |
| 681 | |
| 682 | The example below illustrates the following case : virtualized infrastructures |
| 683 | are deployed in 3 datacenters (DC1..DC3). Each DC uses its own VIP which is |
| 684 | handled by the hosting provider's layer 3 load balancer. This load balancer |
| 685 | routes the traffic to a farm of layer 7 SSL/cache offloaders which load balance |
| 686 | among their local servers. The VIPs are advertised by geolocalised DNS so that |
| 687 | clients generally stick to a given DC. Since clients are not guaranteed to |
| 688 | stick to one DC, the L7 load balancing proxies have to know the other DCs' |
| 689 | servers that may be reached via the hosting provider's LAN or via the internet. |
| 690 | The L7 proxies use the PROXY protocol to join the servers behind them, so that |
| 691 | even inter-DC traffic can forward the original client's address and the return |
| 692 | path is unambiguous. This would not be possible using transparent proxy because |
| 693 | most often the L7 proxies would not be able to spoof an address, and this would |
| 694 | never work between datacenters. |
| 695 | |
| 696 | Internet |
| 697 | |
| 698 | DC1 DC2 DC3 |
| 699 | ,---. ,---. ,---. |
| 700 | ( X ) ( X ) ( X ) |
| 701 | `---' `---' `---' |
| 702 | | +-------+ | +-------+ | +-------+ |
| 703 | +----| L3 LB | +----| L3 LB | +----| L3 LB | |
| 704 | | +-------+ | +-------+ | +-------+ |
| 705 | ------+------- ~ ~ ~ ------+------- ~ ~ ~ ------+------- |
| 706 | ||||| |||| ||||| |||| ||||| |||| |
| 707 | 50 SRV 4 PX 50 SRV 4 PX 50 SRV 4 PX |
| 708 | |
| 709 | |
| 710 | 5. Security considerations |
| 711 | |
| 712 | Version 1 of the protocol header (the human-readable format) was designed so as |
| 713 | to be distinguishable from HTTP. It will not parse as a valid HTTP request and |
| 714 | an HTTP request will not parse as a valid proxy request. Version 2 add to use a |
| 715 | non-parsable binary signature to make many products fail on this block. The |
| 716 | signature was designed to cause immediate failure on HTTP, SSL/TLS, SMTP, FTP, |
| 717 | and POP. It also causes aborts on LDAP and RDP servers (see section 6). That |
| 718 | makes it easier to enforce its use under certain connections and at the same |
| 719 | time, it ensures that improperly configured servers are quickly detected. |
| 720 | |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 721 | Implementers should be very careful about not trying to automatically detect |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 722 | whether they have to decode the header or not, but rather they must only rely |
| 723 | on a configuration parameter. Indeed, if the opportunity is left to a normal |
| 724 | client to use the protocol, he will be able to hide his activities or make them |
| 725 | appear as coming from someone else. However, accepting the header only from a |
| 726 | number of known sources should be safe. |
| 727 | |
| 728 | |
| 729 | 6. Validation |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 730 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 731 | The version 2 protocol signature has been sent to a wide variety of protocols |
| 732 | and implementations including old ones. The following protocol and products |
| 733 | have been tested to ensure the best possible behaviour when the signature was |
| 734 | presented, even with minimal implementations : |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 735 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 736 | - HTTP : |
| 737 | - Apache 1.3.33 : connection abort => pass/optimal |
| 738 | - Nginx 0.7.69 : 400 Bad Request + abort => pass/optimal |
| 739 | - lighttpd 1.4.20 : 400 Bad Request + abort => pass/optimal |
| 740 | - thttpd 2.20c : 400 Bad Request + abort => pass/optimal |
| 741 | - mini-httpd-1.19 : 400 Bad Request + abort => pass/optimal |
| 742 | - haproxy 1.4.21 : 400 Bad Request + abort => pass/optimal |
Willy Tarreau | a124eb6 | 2014-07-12 17:31:07 +0200 | [diff] [blame] | 743 | - Squid 3 : 400 Bad Request + abort => pass/optimal |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 744 | - SSL : |
| 745 | - stud 0.3.47 : connection abort => pass/optimal |
| 746 | - stunnel 4.45 : connection abort => pass/optimal |
| 747 | - nginx 0.7.69 : 400 Bad Request + abort => pass/optimal |
| 748 | - FTP : |
| 749 | - Pure-ftpd 1.0.20 : 3*500 then 221 Goodbye => pass/optimal |
| 750 | - vsftpd 2.0.1 : 3*530 then 221 Goodbye => pass/optimal |
| 751 | - SMTP : |
| 752 | - postfix 2.3 : 3*500 + 221 Bye => pass/optimal |
| 753 | - exim 4.69 : 554 + connection abort => pass/optimal |
| 754 | - POP : |
| 755 | - dovecot 1.0.10 : 3*ERR + Logout => pass/optimal |
| 756 | - IMAP : |
| 757 | - dovecot 1.0.10 : 5*ERR + hang => pass/non-optimal |
| 758 | - LDAP : |
| 759 | - openldap 2.3 : abort => pass/optimal |
| 760 | - SSH : |
| 761 | - openssh 3.9p1 : abort => pass/optimal |
| 762 | - RDP : |
| 763 | - Windows XP SP3 : abort => pass/optimal |
| 764 | |
| 765 | This means that most protocols and implementations will not be confused by an |
| 766 | incoming connection exhibiting the protocol signature, which avoids issues when |
| 767 | facing misconfigurations. |
| 768 | |
| 769 | |
| 770 | 7. Future developments |
Willy Tarreau | 640cf22 | 2010-10-29 21:46:16 +0200 | [diff] [blame] | 771 | |
| 772 | It is possible that the protocol may slightly evolve to present other |
| 773 | information such as the incoming network interface, or the origin addresses in |
| 774 | case of network address translation happening before the first proxy, but this |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 775 | is not identified as a requirement right now. Some deep thinking has been spent |
| 776 | on this and it appears that trying to add a few more information open a pandora |
| 777 | box with many information from MAC addresses to SSL client certificates, which |
| 778 | would make the protocol much more complex. So at this point it is not planned. |
| 779 | Suggestions on improvements are welcome. |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 780 | |
| 781 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 782 | 8. Contacts and links |
Willy Tarreau | 7f89851 | 2011-03-20 11:32:40 +0100 | [diff] [blame] | 783 | |
| 784 | Please use w@1wt.eu to send any comments to the author. |
| 785 | |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 786 | The following links were referenced in the document. |
| 787 | |
| 788 | [1] http://www.postfix.org/XCLIENT_README.html |
Willy Tarreau | 7a6f134 | 2014-06-14 11:45:09 +0200 | [diff] [blame] | 789 | [2] http://tools.ietf.org/html/rfc7239 |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 790 | [3] http://www.stunnel.org/ |
| 791 | [4] https://github.com/bumptech/stud |
| 792 | [5] https://github.com/bumptech/stud/pull/81 |
Willy Tarreau | 998addf | 2015-05-02 15:13:07 +0200 | [diff] [blame^] | 793 | [6] https://www.varnish-cache.org/docs/trunk/phk/ssl_again.html |
| 794 | [7] http://wiki.squid-cache.org/Squid-3.5 |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 795 | |
| 796 | |
| 797 | 9. Sample code |
| 798 | |
| 799 | The code below is an example of how a receiver may deal with both versions of |
| 800 | the protocol header for TCP over IPv4 or IPv6. The function is supposed to be |
| 801 | called upon a read event. Addresses may be directly copied into their final |
| 802 | memory location since they're transported in network byte order. The sending |
| 803 | side is even simpler and can easily be deduced from this sample code. |
| 804 | |
| 805 | struct sockaddr_storage from; /* already filled by accept() */ |
| 806 | struct sockaddr_storage to; /* already filled by getsockname() */ |
Willy Tarreau | 01320c9 | 2014-06-14 08:36:29 +0200 | [diff] [blame] | 807 | const char v2sig[12] = "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A"; |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 808 | |
| 809 | /* returns 0 if needs to poll, <0 upon error or >0 if it did the job */ |
| 810 | int read_evt(int fd) |
| 811 | { |
| 812 | union { |
| 813 | struct { |
| 814 | char line[108]; |
| 815 | } v1; |
| 816 | struct { |
| 817 | uint8_t sig[12]; |
Willy Tarreau | 0f6093a | 2014-06-11 21:21:26 +0200 | [diff] [blame] | 818 | uint8_t ver_cmd; |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 819 | uint8_t fam; |
Willy Tarreau | 0f6093a | 2014-06-11 21:21:26 +0200 | [diff] [blame] | 820 | uint16_t len; |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 821 | union { |
| 822 | struct { /* for TCP/UDP over IPv4, len = 12 */ |
| 823 | uint32_t src_addr; |
| 824 | uint32_t dst_addr; |
| 825 | uint16_t src_port; |
| 826 | uint16_t dst_port; |
| 827 | } ip4; |
| 828 | struct { /* for TCP/UDP over IPv6, len = 36 */ |
| 829 | uint8_t src_addr[16]; |
| 830 | uint8_t dst_addr[16]; |
| 831 | uint16_t src_port; |
| 832 | uint16_t dst_port; |
| 833 | } ip6; |
| 834 | struct { /* for AF_UNIX sockets, len = 216 */ |
| 835 | uint8_t src_addr[108]; |
| 836 | uint8_t dst_addr[108]; |
| 837 | } unx; |
| 838 | } addr; |
| 839 | } v2; |
| 840 | } hdr; |
| 841 | |
| 842 | int size, ret; |
| 843 | |
| 844 | do { |
| 845 | ret = recv(fd, &hdr, sizeof(hdr), MSG_PEEK); |
| 846 | } while (ret == -1 && errno == EINTR); |
| 847 | |
| 848 | if (ret == -1) |
| 849 | return (errno == EAGAIN) ? 0 : -1; |
| 850 | |
Willy Tarreau | 01320c9 | 2014-06-14 08:36:29 +0200 | [diff] [blame] | 851 | if (ret >= 16 && memcmp(&hdr.v2, v2sig, 12) == 0 && |
| 852 | (hdr.v2.ver_cmd & 0xF0) == 0x20) { |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 853 | size = 16 + hdr.v2.len; |
| 854 | if (ret < size) |
| 855 | return -1; /* truncated or too large header */ |
| 856 | |
Willy Tarreau | 0f6093a | 2014-06-11 21:21:26 +0200 | [diff] [blame] | 857 | switch (hdr.v2.ver_cmd & 0xF) { |
Willy Tarreau | 332d7b0 | 2012-11-19 11:27:29 +0100 | [diff] [blame] | 858 | case 0x01: /* PROXY command */ |
| 859 | switch (hdr.v2.fam) { |
| 860 | case 0x11: /* TCPv4 */ |
| 861 | ((struct sockaddr_in *)&from)->sin_family = AF_INET; |
| 862 | ((struct sockaddr_in *)&from)->sin_addr.s_addr = |
| 863 | hdr.v2.addr.ip4.src_addr; |
| 864 | ((struct sockaddr_in *)&from)->sin_port = |
| 865 | hdr.v2.addr.ip4.src_port; |
| 866 | ((struct sockaddr_in *)&to)->sin_family = AF_INET; |
| 867 | ((struct sockaddr_in *)&to)->sin_addr.s_addr = |
| 868 | hdr.v2.addr.ip4.dst_addr; |
| 869 | ((struct sockaddr_in *)&to)->sin_port = |
| 870 | hdr.v2.addr.ip4.dst_port; |
| 871 | goto done; |
| 872 | case 0x21: /* TCPv6 */ |
| 873 | ((struct sockaddr_in6 *)&from)->sin6_family = AF_INET6; |
| 874 | memcpy(&((struct sockaddr_in6 *)&from)->sin6_addr, |
| 875 | hdr.v2.addr.ip6.src_addr, 16); |
| 876 | ((struct sockaddr_in6 *)&from)->sin6_port = |
| 877 | hdr.v2.addr.ip6.src_port; |
| 878 | ((struct sockaddr_in6 *)&to)->sin6_family = AF_INET6; |
| 879 | memcpy(&((struct sockaddr_in6 *)&to)->sin6_addr, |
| 880 | hdr.v2.addr.ip6.dst_addr, 16); |
| 881 | ((struct sockaddr_in6 *)&to)->sin6_port = |
| 882 | hdr.v2.addr.ip6.dst_port; |
| 883 | goto done; |
| 884 | } |
| 885 | /* unsupported protocol, keep local connection address */ |
| 886 | break; |
| 887 | case 0x00: /* LOCAL command */ |
| 888 | /* keep local connection address for LOCAL */ |
| 889 | break; |
| 890 | default: |
| 891 | return -1; /* not a supported command */ |
| 892 | } |
| 893 | } |
| 894 | else if (ret >= 8 && memcmp(hdr.v1.line, "PROXY", 5) == 0) { |
| 895 | char *end = memchr(hdr.v1.line, '\r', ret - 1); |
| 896 | if (!end || end[1] != '\n') |
| 897 | return -1; /* partial or invalid header */ |
| 898 | *end = '\0'; /* terminate the string to ease parsing */ |
| 899 | size = end + 2 - hdr.v1.line; /* skip header + CRLF */ |
| 900 | /* parse the V1 header using favorite address parsers like inet_pton. |
| 901 | * return -1 upon error, or simply fall through to accept. |
| 902 | */ |
| 903 | } |
| 904 | else { |
| 905 | /* Wrong protocol */ |
| 906 | return -1; |
| 907 | } |
| 908 | |
| 909 | done: |
| 910 | /* we need to consume the appropriate amount of data from the socket */ |
| 911 | do { |
| 912 | ret = recv(fd, &hdr, size, 0); |
| 913 | } while (ret == -1 && errno == EINTR); |
| 914 | return (ret >= 0) ? 1 : -1; |
| 915 | } |