blob: a2dbceaf5d34fd3074e8340f8e1c8f97a042f2d0 [file] [log] [blame]
Willy Tarreau7a6f1342014-06-14 11:45:09 +020012014/06/14 Willy Tarreau
Willy Tarreaua3393952014-05-10 15:16:43 +02002 HAProxy Technologies
Willy Tarreau7f898512011-03-20 11:32:40 +01003 The PROXY protocol
Willy Tarreau332d7b02012-11-19 11:27:29 +01004 Versions 1 & 2
Willy Tarreau7f898512011-03-20 11:32:40 +01005
6Abstract
7
8 The PROXY protocol provides a convenient way to safely transport connection
9 information such as a client's address across multiple layers of NAT or TCP
10 proxies. It is designed to require little changes to existing components and
11 to limit the performance impact caused by the processing of the transported
12 information.
13
14
15Revision history
16
17 2010/10/29 - first version
18 2011/03/20 - update: implementation and security considerations
Willy Tarreau332d7b02012-11-19 11:27:29 +010019 2012/06/21 - add support for binary format
20 2012/11/19 - final review and fixes
David Safb76832014-05-08 23:42:08 -040021 2014/05/18 - modify and extend PROXY protocol version 2
Willy Tarreau7a6f1342014-06-14 11:45:09 +020022 2014/06/11 - fix example code to consider ver+cmd merge
23 2014/06/14 - fix v2 header check in example code, and update Forwarded spec
Willy Tarreau7f898512011-03-20 11:32:40 +010024
25
261. Background
Willy Tarreau640cf222010-10-29 21:46:16 +020027
28Relaying TCP connections through proxies generally involves a loss of the
29original TCP connection parameters such as source and destination addresses,
30ports, and so on. Some protocols make it a little bit easier to transfer such
Willy Tarreau332d7b02012-11-19 11:27:29 +010031information. For SMTP, Postfix authors have proposed the XCLIENT protocol [1]
Willy Tarreau7a6f1342014-06-14 11:45:09 +020032which received broad adoption and is particularly suited to mail exchanges.
33For HTTP, there is the "Forwarded" extension [2], which aims at replacing the
34omnipresent "X-Forwarded-For" header which carries information about the
35original source address, and the less common X-Original-To which carries
36information about the destination address.
Willy Tarreau640cf222010-10-29 21:46:16 +020037
38However, both mechanisms require a knowledge of the underlying protocol to be
39implemented in intermediaries.
40
41Then comes a new class of products which we'll call "dumb proxies", not because
42they don't do anything, but because they're processing protocol-agnostic data.
Willy Tarreau332d7b02012-11-19 11:27:29 +010043Both Stunnel[3] and Stud[4] are examples of such "dumb proxies". They talk raw
44TCP on one side, and raw SSL on the other one, and do that reliably, without
Willy Tarreau7a6f1342014-06-14 11:45:09 +020045any knowledge of what protocol is transported on top of the connection. Haproxy
46running in pure TCP mode obviously falls into that category as well.
Willy Tarreau640cf222010-10-29 21:46:16 +020047
48The problem with such a proxy when it is combined with another one such as
Willy Tarreau7a6f1342014-06-14 11:45:09 +020049haproxy, is to adapt it to talk the higher level protocol. A patch is available
Willy Tarreau332d7b02012-11-19 11:27:29 +010050for Stunnel to make it capable of inserting an X-Forwarded-For header in the
51first HTTP request of each incoming connection. Haproxy is able not to add
52another one when the connection comes from Stunnel, so that it's possible to
53hide it from the servers.
Willy Tarreau640cf222010-10-29 21:46:16 +020054
55The typical architecture becomes the following one :
56
57
58 +--------+ HTTP :80 +----------+
59 | client | --------------------------------> | |
60 | | | haproxy, |
61 +--------+ +---------+ | 1 or 2 |
62 / / HTTPS | stunnel | HTTP :81 | listening|
63 <________/ ---------> | (server | ---------> | ports |
64 | mode) | | |
65 +---------+ +----------+
66
67
68The problem appears when haproxy runs with keep-alive on the side towards the
69client. The Stunnel patch will only add the X-Forwarded-For header to the first
70request of each connection and all subsequent requests will not have it. One
71solution could be to improve the patch to make it support keep-alive and parse
72all forwarded data, whether they're announced with a Content-Length or with a
73Transfer-Encoding, taking care of special methods such as HEAD which announce
74data without transfering them, etc... In fact, it would require implementing a
75full HTTP stack in Stunnel. It would then become a lot more complex, a lot less
76reliable and would not anymore be the "dumb proxy" that fits every purposes.
77
78In practice, we don't need to add a header for each request because we'll emit
79the exact same information every time : the information related to the client
80side connection. We could then cache that information in haproxy and use it for
81every other request. But that becomes dangerous and is still limited to HTTP
82only.
83
Willy Tarreau332d7b02012-11-19 11:27:29 +010084Another approach consists in prepending each connection with a header reporting
85the characteristics of the other side's connection. This method is simpler to
Willy Tarreau640cf222010-10-29 21:46:16 +020086implement, does not require any protocol-specific knowledge on either side, and
Willy Tarreau332d7b02012-11-19 11:27:29 +010087completely fits the purpose since what is desired precisely is to know the
88other side's connection endpoints. It is easy to perform for the sender (just
89send a short header once the connection is established) and to parse for the
90receiver (simply perform one read() on the incoming connection to fill in
91addresses after an accept). The protocol used to carry connection information
92across proxies was thus called the PROXY protocol.
Willy Tarreau640cf222010-10-29 21:46:16 +020093
Willy Tarreau7f898512011-03-20 11:32:40 +010094
Willy Tarreau332d7b02012-11-19 11:27:29 +0100952. The PROXY protocol header
Willy Tarreau7f898512011-03-20 11:32:40 +010096
Willy Tarreau332d7b02012-11-19 11:27:29 +010097This document uses a few terms that are worth explaining here :
98 - "connection initiator" is the party requesting a new connection
99 - "connection target" is the party accepting a connection request
100 - "client" is the party for which a connection was requested
101 - "server" is the party to which the client desired to connect
102 - "proxy" is the party intercepting and relaying the connection
103 from the client to the server.
104 - "sender" is the party sending data over a connection.
105 - "receiver" is the party receiving data from the sender.
106 - "header" or "PROXY protocol header" is the block of connection information
107 the connection initiator prepends at the beginning of a connection, which
108 makes it the sender from the protocol point of view.
109
110The PROXY protocol's goal is to fill the server's internal structures with the
111information collected by the proxy that the server would have been able to get
112by itself if the client was connecting directly to the server instead of via a
113proxy. The information carried by the protocol are the ones the server would
114get using getsockname() and getpeername() :
115 - address family (AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX)
116 - socket protocol (SOCK_STREAM for TCP, SOCK_DGRAM for UDP)
Willy Tarreau640cf222010-10-29 21:46:16 +0200117 - layer 3 source and destination addresses
118 - layer 4 source and destination ports if any
119
120Unlike the XCLIENT protocol, the PROXY protocol was designed with limited
Willy Tarreau332d7b02012-11-19 11:27:29 +0100121extensibility in order to help the receiver parse it very fast. Version 1 was
122focused on keeping it human-readable for better debugging possibilities, which
123is always desirable for early adoption when few implementations exist. Version
1242 adds support for a binary encoding of the header which is much more efficient
125to produce and to parse, especially when dealing with IPv6 addresses that are
126expensive to emit in ASCII form and to parse.
127
128In both cases, the protocol simply consists in an easily parsable header placed
129by the connection initiator at the beginning of each connection. The protocol
130is intentionally stateless in that it does not expect the sender to wait for
131the receiver before sending the header, nor the receiver to send anything back.
132
133This specification supports two header formats, a human-readable format which
134is the only format supported in version 1 of the protocol, and a binary format
135which is only supported in version 2. Both formats were designed to ensure that
136the header cannot be confused with common higher level protocols such as HTTP,
137SSL/TLS, FTP or SMTP, and that both formats are easily distinguishable one from
138each other for the receiver.
139
140Version 1 senders MAY only produce the human-readable header format. Version 2
141senders MAY only produce the binary header format. Version 1 receivers MUST at
142least implement the human-readable header format. Version 2 receivers MUST at
143least implement the binary header format, and it is recommended that they also
144implement the human-readable header format for better interoperability and ease
145of upgrade when facing version 1 senders.
146
147Both formats are designed to fit in the smallest TCP segment that any TCP/IP
148host is required to support (576 - 40 = 536 bytes). This ensures that the whole
149header will always be delivered at once when the socket buffers are still empty
150at the beginning of a connection. The sender must always ensure that the header
151is sent at once, so that the transport layer maintains atomicity along the path
152to the receiver. The receiver may be tolerant to partial headers or may simply
153drop the connection when receiving a partial header. Recommendation is to be
154tolerant, but implementation constraints may not always easily permit this. It
155is important to note that nothing forces any intermediary to forward the whole
156header at once, because TCP is a streaming protocol which may be processed one
157byte at a time if desired, causing the header to be fragmented when reaching
158the receiver. But due to the places where such a protocol is used, the above
159simplification generally is acceptable because the risk of crossing such a
160device handling one byte at a time is close to zero.
161
162The receiver MUST NOT start processing the connection before it receives a
163complete and valid PROXY protocol header. This is particularly important for
164protocols where the receiver is expected to speak first (eg: SMTP, FTP or SSH).
165The receiver may apply a short timeout and decide to abort the connection if
166the protocol header is not seen within a few seconds (at least 3 seconds to
167cover a TCP retransmit).
168
169The receiver MUST be configured to only receive the protocol described in this
170specification and MUST not try to guess whether the protocol header is present
171or not. This means that the protocol explicitly prevents port sharing between
172public and private access. Otherwise it would open a major security breach by
173allowing untrusted parties to spoof their connection addresses. The receiver
174SHOULD ensure proper access filtering so that only trusted proxies are allowed
175to use this protocol.
176
177Some proxies are smart enough to understand transported protocols and to reuse
178idle server connections for multiple messages. This typically happens in HTTP
179where requests from multiple clients may be sent over the same connection. Such
180proxies MUST NOT implement this protocol on multiplexed connections because the
181receiver would use the address advertised in the PROXY header as the address of
182all forwarded requests's senders. In fact, such proxies are not dumb proxies,
183and since they do have a complete understanding of the transported protocol,
184they MUST use the facilities provided by this protocol to present the client's
185address.
186
187
1882.1. Human-readable header format (Version 1)
189
190This is the format specified in version 1 of the protocol. It consists in one
191line of ASCII text matching exactly the following block, sent immediately and
192at once upon the connection establishment and prepended before any data flowing
193from the sender to the receiver :
Willy Tarreau640cf222010-10-29 21:46:16 +0200194
195 - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 )
Willy Tarreau332d7b02012-11-19 11:27:29 +0100196 Seeing this string indicates that this is version 1 of the protocol.
Willy Tarreau640cf222010-10-29 21:46:16 +0200197
198 - exactly one space : " " ( \x20 )
199
Willy Tarreau332d7b02012-11-19 11:27:29 +0100200 - a string indicating the proxied INET protocol and family. As of version 1,
Willy Tarreau640cf222010-10-29 21:46:16 +0200201 only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6"
Willy Tarreau332d7b02012-11-19 11:27:29 +0100202 ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Other, unsupported,
203 or unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E
204 \x4B \x4E \x4F \x57 \x4E ). For "UNKNOWN", the rest of the line before the
205 CRLF may be omitted by the sender, and the receiver must ignore anything
206 presented before the CRLF is found. Note that an earlier version of this
207 specification suggested to use this when sending health checks, but this
208 causes issues with servers that reject the "UNKNOWN" keyword. Thus is it
209 now recommended not to send "UNKNOWN" when the connection is expected to
210 be accepted, but only when it is not possible to correctly fill the PROXY
211 line.
Willy Tarreau640cf222010-10-29 21:46:16 +0200212
213 - exactly one space : " " ( \x20 )
214
215 - the layer 3 source address in its canonical format. IPv4 addresses must be
216 indicated as a series of exactly 4 integers in the range [0..255] inclusive
217 written in decimal representation separated by exactly one dot between each
218 other. Heading zeroes are not permitted in front of numbers in order to
219 avoid any possible confusion with octal numbers. IPv6 addresses must be
220 indicated as series of 4 hexadecimal digits (upper or lower case) delimited
221 by colons between each other, with the acceptance of one double colon
222 sequence to replace the largest acceptable range of consecutive zeroes. The
223 total number of decoded bits must exactly be 128. The advertised protocol
224 family dictates what format to use.
225
226 - exactly one space : " " ( \x20 )
227
228 - the layer 3 destination address in its canonical format. It is the same
229 format as the layer 3 source address and matches the same family.
230
231 - exactly one space : " " ( \x20 )
232
233 - the TCP source port represented as a decimal integer in the range
234 [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
235 in order to avoid any possible confusion with octal numbers.
236
237 - exactly one space : " " ( \x20 )
238
239 - the TCP destination port represented as a decimal integer in the range
240 [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
241 in order to avoid any possible confusion with octal numbers.
242
243 - the CRLF sequence ( \x0D \x0A )
244
Willy Tarreau332d7b02012-11-19 11:27:29 +0100245
246The maximum line lengths the receiver must support including the CRLF are :
247 - TCP/IPv4 :
248 "PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n"
249 => 5 + 1 + 4 + 1 + 15 + 1 + 15 + 1 + 5 + 1 + 5 + 2 = 56 chars
250
251 - TCP/IPv6 :
252 "PROXY TCP6 ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n"
253 => 5 + 1 + 4 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 104 chars
254
255 - unknown connection (short form) :
256 "PROXY UNKNOWN\r\n"
257 => 5 + 1 + 7 + 2 = 15 chars
258
259 - worst case (optional fields set to 0xff) :
260 "PROXY UNKNOWN ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n"
261 => 5 + 1 + 7 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 107 chars
262
263So a 108-byte buffer is always enough to store all the line and a trailing zero
264for string processing.
265
266The receiver must wait for the CRLF sequence before starting to decode the
267addresses in order to ensure they are complete and properly parsed. If the CRLF
268sequence is not found in the first 107 characters, the receiver should declare
269the line invalid. A receiver may reject an incomplete line which does not
270contain the CRLF sequence in the first atomic read operation. The receiver must
271not tolerate a single CR or LF character to end the line when a complete CRLF
272sequence is expected.
273
274Any sequence which does not exactly match the protocol must be discarded and
275cause the receiver to abort the connection. It is recommended to abort the
276connection as soon as possible so that the sender gets a chance to notice the
277anomaly and log it.
Willy Tarreau640cf222010-10-29 21:46:16 +0200278
279If the announced transport protocol is "UNKNOWN", then the receiver knows that
Willy Tarreau332d7b02012-11-19 11:27:29 +0100280the sender speaks the correct PROXY protocol with the appropriate version, and
281SHOULD accept the connection and use the real connection's parameters as if
282there were no PROXY protocol header on the wire. However, senders SHOULD not
283use the "UNKNOWN" protocol when they are the initiators of outgoing connections
284because some receivers may reject them. When a load balancing proxy has to send
285health checks to a server, it SHOULD build a valid PROXY line which it will
286fill with a getsockname()/getpeername() pair indicating the addresses used. It
287is important to understand that doing so is not appropriate when some source
288address translation is performed between the sender and the receiver.
Willy Tarreau640cf222010-10-29 21:46:16 +0200289
290An example of such a line before an HTTP request would look like this (CR
291marked as "\r" and LF marked as "\n") :
292
293 PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
294 GET / HTTP/1.1\r\n
295 Host: 192.168.0.11\r\n
296 \r\n
297
Willy Tarreau332d7b02012-11-19 11:27:29 +0100298For the sender, the header line is easy to put into the output buffers once the
299connection is established. Note that since the line is always shorter than an
300MSS, the sender is guaranteed to always be able to emit it at once and should
301not even bother handling partial sends. For the receiver, once the header is
302parsed, it is easy to skip it from the input buffers. Please consult section 9
303for implementation suggestions.
304
305
3062.2. Binary header format (version 2)
307
308Producing human-readable IPv6 addresses and parsing them is very inefficient,
309due to the multiple possible representation formats and the handling of compact
310address format. It was also not possible to specify address families outside
311IPv4/IPv6 nor non-TCP protocols. Another drawback of the human-readable format
312is the fact that implementations need to parse all characters to find the
313trailing CRLF, which makes it harder to read only the exact bytes count. Last,
314the UNKNOWN address type has not always been accepted by servers as a valid
315protocol because of its imprecise meaning.
316
317Version 2 of the protocol thus introduces a new binary format which remains
318distinguishable from version 1 and from other commonly used protocols. It was
319specially designed in order to be incompatible with a wide range of protocols
320and to be rejected by a number of common implementations of these protocols
321when unexpectedly presented (please see section 7). Also for better processing
322efficiency, IPv4 and IPv6 addresses are respectively aligned on 4 and 16 bytes
323boundaries.
324
325The binary header format starts with a constant 12 bytes block containing the
326protocol signature :
327
328 \x0D \x0A \x0D \x0A \x00 \x0D \x0A \x51 \x55 \x49 \x54 \x0A
329
330Note that this block contains a null byte at the 5th position, so it must not
331be handled as a null-terminated string.
332
David Safb76832014-05-08 23:42:08 -0400333The next byte (the 13th one) is the protocol version and command.
Willy Tarreau332d7b02012-11-19 11:27:29 +0100334
David Safb76832014-05-08 23:42:08 -0400335The highest four bits contains the version. As of this specification, it must
336always be sent as \x2 and the receiver must only accept this value.
337
338The lowest four bits represents the command :
339 - \x0 : LOCAL : the connection was established on purpose by the proxy
Willy Tarreau332d7b02012-11-19 11:27:29 +0100340 without being relayed. The connection endpoints are the sender and the
341 receiver. Such connections exist when the proxy sends health-checks to the
342 server. The receiver must accept this connection as valid and must use the
343 real connection endpoints and discard the protocol block including the
344 family which is ignored.
345
David Safb76832014-05-08 23:42:08 -0400346 - \x1 : PROXY : the connection was established on behalf of another node,
Willy Tarreau332d7b02012-11-19 11:27:29 +0100347 and reflects the original connection endpoints. The receiver must then use
348 the information provided in the protocol block to get original the address.
349
350 - other values are unassigned and must not be emitted by senders. Receivers
351 must drop connections presenting unexpected values here.
352
David Safb76832014-05-08 23:42:08 -0400353The 14th byte contains the transport protocol and address family. The highest 4
Willy Tarreau332d7b02012-11-19 11:27:29 +0100354bits contain the address family, the lowest 4 bits contain the protocol.
355
356The address family maps to the original socket family without necessarily
357matching the values internally used by the system. It may be one of :
358
359 - 0x0 : AF_UNSPEC : the connection is forwarded for an unknown, unspecified
360 or unsupported protocol. The sender should use this family when sending
361 LOCAL commands or when dealing with unsupported protocol families. The
362 receiver is free to accept the connection anyway and use the real endpoint
363 addresses or to reject it. The receiver should ignore address information.
364
365 - 0x1 : AF_INET : the forwarded connection uses the AF_INET address family
366 (IPv4). The addresses are exactly 4 bytes each in network byte order,
367 followed by transport protocol information (typically ports).
368
369 - 0x2 : AF_INET6 : the forwarded connection uses the AF_INET6 address family
370 (IPv6). The addresses are exactly 16 bytes each in network byte order,
371 followed by transport protocol information (typically ports).
372
373 - 0x3 : AF_UNIX : the forwarded connection uses the AF_UNIX address family
374 (UNIX). The addresses are exactly 108 bytes each.
375
376 - other values are unspecified and must not be emitted in version 2 of this
377 protocol and must be rejected as invalid by receivers.
378
David Safb76832014-05-08 23:42:08 -0400379The transport protocol is specified in the lowest 4 bits of the the 14th byte :
Willy Tarreau332d7b02012-11-19 11:27:29 +0100380
381 - 0x0 : UNSPEC : the connection is forwarded for an unknown, unspecified
382 or unsupported protocol. The sender should use this family when sending
383 LOCAL commands or when dealing with unsupported protocol families. The
384 receiver is free to accept the connection anyway and use the real endpoint
385 addresses or to reject it. The receiver should ignore address information.
386
387 - 0x1 : STREAM : the forwarded connection uses a SOCK_STREAM protocol (eg:
388 TCP or UNIX_STREAM). When used with AF_INET/AF_INET6 (TCP), the addresses
389 are followed by the source and destination ports represented on 2 bytes
390 each in network byte order.
391
392 - 0x2 : DGRAM : the forwarded connection uses a SOCK_DGRAM protocol (eg:
393 UDP or UNIX_DGRAM). When used with AF_INET/AF_INET6 (UDP), the addresses
394 are followed by the source and destination ports represented on 2 bytes
395 each in network byte order.
396
397 - other values are unspecified and must not be emitted in version 2 of this
398 protocol and must be rejected as invalid by receivers.
399
400In practice, the following protocol bytes are expected :
401
402 - \x00 : UNSPEC : the connection is forwarded for an unknown, unspecified
403 or unsupported protocol. The sender should use this family when sending
404 LOCAL commands or when dealing with unsupported protocol families. When
405 used with a LOCAL command, the receiver must accept the connection and
406 ignore any address information. For other commands, the receiver is free
407 to accept the connection anyway and use the real endpoints addresses or to
408 reject the connection. The receiver should ignore address information.
409
410 - \x11 : TCP over IPv4 : the forwarded connection uses TCP over the AF_INET
411 protocol family. Address length is 2*4 + 2*2 = 12 bytes.
412
413 - \x12 : UDP over IPv4 : the forwarded connection uses UDP over the AF_INET
414 protocol family. Address length is 2*4 + 2*2 = 12 bytes.
415
416 - \x21 : TCP over IPv6 : the forwarded connection uses TCP over the AF_INET6
417 protocol family. Address length is 2*16 + 2*2 = 36 bytes.
418
419 - \x22 : UDP over IPv6 : the forwarded connection uses UDP over the AF_INET6
420 protocol family. Address length is 2*16 + 2*2 = 36 bytes.
421
422 - \x31 : UNIX stream : the forwarded connection uses SOCK_STREAM over the
423 AF_UNIX protocol family. Address length is 2*108 = 216 bytes.
424
425 - \x32 : UNIX datagram : the forwarded connection uses SOCK_DGRAM over the
426 AF_UNIX protocol family. Address length is 2*108 = 216 bytes.
427
428
429Only the UNSPEC protocol byte (\x00) is mandatory. A receiver is not required
430to implement other ones, provided that it automatically falls back to the
431UNSPEC mode for the valid combinations above that it does not support.
Willy Tarreau640cf222010-10-29 21:46:16 +0200432
David Safb76832014-05-08 23:42:08 -0400433The 15th and 16th bytes is the address length in bytes in network endien order.
434It is used so that the receiver knows how many address bytes to skip even when
435it does not implement the presented protocol. Thus the length of the protocol
436header in bytes is always exactly 16 + this value. When a sender presents a
Willy Tarreau332d7b02012-11-19 11:27:29 +0100437LOCAL connection, it should not present any address so it sets this field to
438zero. Receivers MUST always consider this field to skip the appropriate number
439of bytes and must not assume zero is presented for LOCAL connections. When a
440receiver accepts an incoming connection showing an UNSPEC address family or
441protocol, it may or may not decide to log the address information if present.
442
443So the 16-byte version 2 header can be described this way :
444
445 struct proxy_hdr_v2 {
446 uint8_t sig[12]; /* hex 0D 0A 0D 0A 00 0D 0A 51 55 49 54 0A */
Willy Tarreau0f6093a2014-06-11 21:21:26 +0200447 uint8_t ver_cmd; /* protocol version and command */
Willy Tarreau332d7b02012-11-19 11:27:29 +0100448 uint8_t fam; /* protocol family and address */
David Safb76832014-05-08 23:42:08 -0400449 uint16_t len; /* number of following bytes part of the header */
Willy Tarreau332d7b02012-11-19 11:27:29 +0100450 };
451
452Starting from the 17th byte, addresses are presented in network byte order.
453The address order is always the same :
454 - source layer 3 address in network byte order
455 - destination layer 3 address in network byte order
456 - source layer 4 address if any, in network byte order (port)
457 - destination layer 4 address if any, in network byte order (port)
458
459The address block may directly be sent from or received into the following
460union which makes it easy to cast from/to the relevant socket native structs
461depending on the address type :
462
463 union proxy_addr {
464 struct { /* for TCP/UDP over IPv4, len = 12 */
465 uint32_t src_addr;
466 uint32_t dst_addr;
467 uint16_t src_port;
468 uint16_t dst_port;
469 } ipv4_addr;
470 struct { /* for TCP/UDP over IPv6, len = 36 */
471 uint8_t src_addr[16];
472 uint8_t dst_addr[16];
473 uint16_t src_port;
474 uint16_t dst_port;
475 } ipv6_addr;
476 struct { /* for AF_UNIX sockets, len = 216 */
477 uint8_t src_addr[108];
478 uint8_t dst_addr[108];
479 } unix_addr;
480 };
481
482The sender must ensure that all the protocol header is sent at once. This block
483is always smaller than an MSS, so there is no reason for it to be segmented at
484the beginning of the connection. The receiver should also process the header
485at once. The receiver must not start to parse an address before the whole
486address block is received. The receiver must also reject incoming connections
487containing partial protocol headers.
488
489A receiver may be configured to support both version 1 and version 2 of the
490protocol. Identifying the protocol version is easy :
491
492 - if the incoming byte count is 16 or above and the 13 first bytes match
493 the protocol signature block followed by the protocol version 2 :
494
495 \x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A\x02
496
497 - otherwise, if the incoming byte count is 8 or above, and the 5 first
498 characters match the ASCII representation of "PROXY" then the protocol
499 must be parsed as version 1 :
500
501 \x50\x52\x4F\x58\x59
502
503 - otherwise the protocol is not covered by this specification and the
504 connection must be dropped.
505
David Safb76832014-05-08 23:42:08 -0400506If the length specified in the PROXY protocol header indicates that additional
507bytes are part of the header beyond the address information, a receiver may
508choose to skip over and ignore those bytes, or attempt to interpret those
509bytes.
510
511The information in those bytes will be arranged in Type-Length-Value (TLV
512vectors) in the following format. The first byte is the Type of the vector.
513The second two bytes represent the length in bytes of the value (not included
514the Type and Length bytes), and following the length field is the number of
515bytes specified by the length.
516
517 struct {
518 uint8_t type;
519 uint8_t length_hi;
520 uint8_t length_lo;
521 uint8_t value[0];
522 } tlv;
523
Willy Tarreau7f898512011-03-20 11:32:40 +0100524
5253. Implementations
526
Willy Tarreau332d7b02012-11-19 11:27:29 +0100527Haproxy 1.5 implements version 1 of the PROXY protocol on both sides :
Willy Tarreau7f898512011-03-20 11:32:40 +0100528 - the listening sockets accept the protocol when the "accept-proxy" setting
529 is passed to the "bind" keyword. Connections accepted on such listeners
530 will behave just as if the source really was the one advertised in the
531 protocol. This is true for logging, ACLs, content filtering, transparent
532 proxying, etc...
533
534 - the protocol may be used to connect to servers if the "send-proxy" setting
535 is present on the "server" line. It is enabled on a per-server basis, so it
536 is possible to have it enabled for remote servers only and still have local
537 ones behave differently. If the incoming connection was accepted with the
538 "accept-proxy", then the relayed information is the one advertised in this
539 connection's PROXY line.
540
David Safb76832014-05-08 23:42:08 -0400541 - Haproxy 1.5 also implements version 2 of the PROXY protocol as a sender. In
542 addition, a TLV with limited, optional, SSL information has been added.
543
Willy Tarreau332d7b02012-11-19 11:27:29 +0100544Stunnel added support for version 1 of the protocol for outgoing connections in
545version 4.45.
Willy Tarreau7f898512011-03-20 11:32:40 +0100546
Willy Tarreau332d7b02012-11-19 11:27:29 +0100547Stud added support for version 1 of the protocol for outgoing connections on
5482011/06/29.
549
550Postfix added support for version 1 of the protocol for incoming connections
551in smtpd and postscreen in version 2.10.
552
553A patch is available for Stud[5] to implement version 1 of the protocol on
554incoming connections.
555
556Support for the protocol in the Varnish cache is being considered [6].
557
Todd Lyonsd1dcea02014-06-03 13:29:33 -0700558Exim added support for version 1 and version 2 of the protocol for incoming
559connections on 2014/05/13, and will be released as part of version 4.83.
560
Willy Tarreau332d7b02012-11-19 11:27:29 +0100561The protocol is simple enough that it is expected that other implementations
562will appear, especially in environments such as SMTP, IMAP, FTP, RDP where the
Willy Tarreau7f898512011-03-20 11:32:40 +0100563client's address is an important piece of information for the server and some
Willy Tarreau332d7b02012-11-19 11:27:29 +0100564intermediaries. In fact, several proprietary deployments have already done so
565on FTP and SMTP servers.
Willy Tarreau7f898512011-03-20 11:32:40 +0100566
567Proxy developers are encouraged to implement this protocol, because it will
568make their products much more transparent in complex infrastructures, and will
569get rid of a number of issues related to logging and access control.
570
Willy Tarreau332d7b02012-11-19 11:27:29 +0100571
5724. Architectural benefits
5734.1. Multiple layers
574
575Using the PROXY protocol instead of transparent proxy provides several benefits
576in multiple-layer infrastructures. The first immediate benefit is that it
577becomes possible to chain multiple layers of proxies and always present the
578original IP address. for instance, let's consider the following 2-layer proxy
579architecture :
580
581 Internet
582 ,---. | client to PX1:
583 ( X ) | native protocol
584 `---' |
585 | V
586 +--+--+ +-----+
587 | FW1 |------| PX1 |
588 +--+--+ +-----+ | PX1 to PX2: PROXY + native
589 | V
590 +--+--+ +-----+
591 | FW2 |------| PX2 |
592 +--+--+ +-----+ | PX2 to SRV: PROXY + native
593 | V
594 +--+--+
595 | SRV |
596 +-----+
Willy Tarreau7f898512011-03-20 11:32:40 +0100597
Willy Tarreau332d7b02012-11-19 11:27:29 +0100598Firewall FW1 receives traffic from internet-based clients and forwards it to
599reverse-proxy PX1. PX1 adds a PROXY header then forwards to PX2 via FW2. PX2
600is configured to read the PROXY header and to emit it on output. It then joins
601the origin server SRV and presents the original client's address there. Since
602all TCP connections endpoints are real machines and are not spoofed, there is
603no issue for the return traffic to pass via the firewalls and reverse proxies.
604Using transparent proxy, this would be quite difficult because the firewalls
605would have to deal with the client's address coming from the proxies in the DMZ
606and would have to correctly route the return traffic there instead of using the
607default route.
Willy Tarreau7f898512011-03-20 11:32:40 +0100608
Willy Tarreau332d7b02012-11-19 11:27:29 +0100609
6104.2. IPv4 and IPv6 integration
611
612The protocol also eases IPv4 and IPv6 integration : if only the first layer
613(FW1 and PX1) is IPv6-capable, it is still possible to present the original
614client's IPv6 address to the target server eventhough the whole chain is only
615connected via IPv4.
616
617
6184.3. Multiple return paths
619
620When transparent proxy is used, it is not possible to run multiple proxies
621because the return traffic would follow the default route instead of finding
622the proper proxy. Some tricks are sometimes possible using multiple server
623addresses and policy routing but these are very limited.
624
625Using the PROXY protocol, this problem disappears as the servers don't need
626to route to the client, just to the proxy that forwarded the connection. So
627it is perfectly possible to run a proxy farm in front of a very large server
628farm and have it working effortless, even when dealing with multiple sites.
629
630This is particularly important in Cloud-like environments where there is little
631choice of binding to random addresses and where the lower processing power per
632node generally requires multiple front nodes.
633
634The example below illustrates the following case : virtualized infrastructures
635are deployed in 3 datacenters (DC1..DC3). Each DC uses its own VIP which is
636handled by the hosting provider's layer 3 load balancer. This load balancer
637routes the traffic to a farm of layer 7 SSL/cache offloaders which load balance
638among their local servers. The VIPs are advertised by geolocalised DNS so that
639clients generally stick to a given DC. Since clients are not guaranteed to
640stick to one DC, the L7 load balancing proxies have to know the other DCs'
641servers that may be reached via the hosting provider's LAN or via the internet.
642The L7 proxies use the PROXY protocol to join the servers behind them, so that
643even inter-DC traffic can forward the original client's address and the return
644path is unambiguous. This would not be possible using transparent proxy because
645most often the L7 proxies would not be able to spoof an address, and this would
646never work between datacenters.
647
648 Internet
649
650 DC1 DC2 DC3
651 ,---. ,---. ,---.
652 ( X ) ( X ) ( X )
653 `---' `---' `---'
654 | +-------+ | +-------+ | +-------+
655 +----| L3 LB | +----| L3 LB | +----| L3 LB |
656 | +-------+ | +-------+ | +-------+
657 ------+------- ~ ~ ~ ------+------- ~ ~ ~ ------+-------
658 ||||| |||| ||||| |||| ||||| ||||
659 50 SRV 4 PX 50 SRV 4 PX 50 SRV 4 PX
660
661
6625. Security considerations
663
664Version 1 of the protocol header (the human-readable format) was designed so as
665to be distinguishable from HTTP. It will not parse as a valid HTTP request and
666an HTTP request will not parse as a valid proxy request. Version 2 add to use a
667non-parsable binary signature to make many products fail on this block. The
668signature was designed to cause immediate failure on HTTP, SSL/TLS, SMTP, FTP,
669and POP. It also causes aborts on LDAP and RDP servers (see section 6). That
670makes it easier to enforce its use under certain connections and at the same
671time, it ensures that improperly configured servers are quickly detected.
672
Willy Tarreau7f898512011-03-20 11:32:40 +0100673Implementers should be very careful about not trying to automatically detect
Willy Tarreau332d7b02012-11-19 11:27:29 +0100674whether they have to decode the header or not, but rather they must only rely
675on a configuration parameter. Indeed, if the opportunity is left to a normal
676client to use the protocol, he will be able to hide his activities or make them
677appear as coming from someone else. However, accepting the header only from a
678number of known sources should be safe.
679
680
6816. Validation
Willy Tarreau7f898512011-03-20 11:32:40 +0100682
Willy Tarreau332d7b02012-11-19 11:27:29 +0100683The version 2 protocol signature has been sent to a wide variety of protocols
684and implementations including old ones. The following protocol and products
685have been tested to ensure the best possible behaviour when the signature was
686presented, even with minimal implementations :
Willy Tarreau7f898512011-03-20 11:32:40 +0100687
Willy Tarreau332d7b02012-11-19 11:27:29 +0100688 - HTTP :
689 - Apache 1.3.33 : connection abort => pass/optimal
690 - Nginx 0.7.69 : 400 Bad Request + abort => pass/optimal
691 - lighttpd 1.4.20 : 400 Bad Request + abort => pass/optimal
692 - thttpd 2.20c : 400 Bad Request + abort => pass/optimal
693 - mini-httpd-1.19 : 400 Bad Request + abort => pass/optimal
694 - haproxy 1.4.21 : 400 Bad Request + abort => pass/optimal
695 - SSL :
696 - stud 0.3.47 : connection abort => pass/optimal
697 - stunnel 4.45 : connection abort => pass/optimal
698 - nginx 0.7.69 : 400 Bad Request + abort => pass/optimal
699 - FTP :
700 - Pure-ftpd 1.0.20 : 3*500 then 221 Goodbye => pass/optimal
701 - vsftpd 2.0.1 : 3*530 then 221 Goodbye => pass/optimal
702 - SMTP :
703 - postfix 2.3 : 3*500 + 221 Bye => pass/optimal
704 - exim 4.69 : 554 + connection abort => pass/optimal
705 - POP :
706 - dovecot 1.0.10 : 3*ERR + Logout => pass/optimal
707 - IMAP :
708 - dovecot 1.0.10 : 5*ERR + hang => pass/non-optimal
709 - LDAP :
710 - openldap 2.3 : abort => pass/optimal
711 - SSH :
712 - openssh 3.9p1 : abort => pass/optimal
713 - RDP :
714 - Windows XP SP3 : abort => pass/optimal
715
716This means that most protocols and implementations will not be confused by an
717incoming connection exhibiting the protocol signature, which avoids issues when
718facing misconfigurations.
719
720
7217. Future developments
Willy Tarreau640cf222010-10-29 21:46:16 +0200722
723It is possible that the protocol may slightly evolve to present other
724information such as the incoming network interface, or the origin addresses in
725case of network address translation happening before the first proxy, but this
Willy Tarreau332d7b02012-11-19 11:27:29 +0100726is not identified as a requirement right now. Some deep thinking has been spent
727on this and it appears that trying to add a few more information open a pandora
728box with many information from MAC addresses to SSL client certificates, which
729would make the protocol much more complex. So at this point it is not planned.
730Suggestions on improvements are welcome.
Willy Tarreau7f898512011-03-20 11:32:40 +0100731
732
Willy Tarreau332d7b02012-11-19 11:27:29 +01007338. Contacts and links
Willy Tarreau7f898512011-03-20 11:32:40 +0100734
735Please use w@1wt.eu to send any comments to the author.
736
Willy Tarreau332d7b02012-11-19 11:27:29 +0100737The following links were referenced in the document.
738
739[1] http://www.postfix.org/XCLIENT_README.html
Willy Tarreau7a6f1342014-06-14 11:45:09 +0200740[2] http://tools.ietf.org/html/rfc7239
Willy Tarreau332d7b02012-11-19 11:27:29 +0100741[3] http://www.stunnel.org/
742[4] https://github.com/bumptech/stud
743[5] https://github.com/bumptech/stud/pull/81
744[6] https://www.varnish-cache.org/trac/wiki/Future_Protocols
745
746
7479. Sample code
748
749The code below is an example of how a receiver may deal with both versions of
750the protocol header for TCP over IPv4 or IPv6. The function is supposed to be
751called upon a read event. Addresses may be directly copied into their final
752memory location since they're transported in network byte order. The sending
753side is even simpler and can easily be deduced from this sample code.
754
755 struct sockaddr_storage from; /* already filled by accept() */
756 struct sockaddr_storage to; /* already filled by getsockname() */
Willy Tarreau01320c92014-06-14 08:36:29 +0200757 const char v2sig[12] = "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A";
Willy Tarreau332d7b02012-11-19 11:27:29 +0100758
759 /* returns 0 if needs to poll, <0 upon error or >0 if it did the job */
760 int read_evt(int fd)
761 {
762 union {
763 struct {
764 char line[108];
765 } v1;
766 struct {
767 uint8_t sig[12];
Willy Tarreau0f6093a2014-06-11 21:21:26 +0200768 uint8_t ver_cmd;
Willy Tarreau332d7b02012-11-19 11:27:29 +0100769 uint8_t fam;
Willy Tarreau0f6093a2014-06-11 21:21:26 +0200770 uint16_t len;
Willy Tarreau332d7b02012-11-19 11:27:29 +0100771 union {
772 struct { /* for TCP/UDP over IPv4, len = 12 */
773 uint32_t src_addr;
774 uint32_t dst_addr;
775 uint16_t src_port;
776 uint16_t dst_port;
777 } ip4;
778 struct { /* for TCP/UDP over IPv6, len = 36 */
779 uint8_t src_addr[16];
780 uint8_t dst_addr[16];
781 uint16_t src_port;
782 uint16_t dst_port;
783 } ip6;
784 struct { /* for AF_UNIX sockets, len = 216 */
785 uint8_t src_addr[108];
786 uint8_t dst_addr[108];
787 } unx;
788 } addr;
789 } v2;
790 } hdr;
791
792 int size, ret;
793
794 do {
795 ret = recv(fd, &hdr, sizeof(hdr), MSG_PEEK);
796 } while (ret == -1 && errno == EINTR);
797
798 if (ret == -1)
799 return (errno == EAGAIN) ? 0 : -1;
800
Willy Tarreau01320c92014-06-14 08:36:29 +0200801 if (ret >= 16 && memcmp(&hdr.v2, v2sig, 12) == 0 &&
802 (hdr.v2.ver_cmd & 0xF0) == 0x20) {
Willy Tarreau332d7b02012-11-19 11:27:29 +0100803 size = 16 + hdr.v2.len;
804 if (ret < size)
805 return -1; /* truncated or too large header */
806
Willy Tarreau0f6093a2014-06-11 21:21:26 +0200807 switch (hdr.v2.ver_cmd & 0xF) {
Willy Tarreau332d7b02012-11-19 11:27:29 +0100808 case 0x01: /* PROXY command */
809 switch (hdr.v2.fam) {
810 case 0x11: /* TCPv4 */
811 ((struct sockaddr_in *)&from)->sin_family = AF_INET;
812 ((struct sockaddr_in *)&from)->sin_addr.s_addr =
813 hdr.v2.addr.ip4.src_addr;
814 ((struct sockaddr_in *)&from)->sin_port =
815 hdr.v2.addr.ip4.src_port;
816 ((struct sockaddr_in *)&to)->sin_family = AF_INET;
817 ((struct sockaddr_in *)&to)->sin_addr.s_addr =
818 hdr.v2.addr.ip4.dst_addr;
819 ((struct sockaddr_in *)&to)->sin_port =
820 hdr.v2.addr.ip4.dst_port;
821 goto done;
822 case 0x21: /* TCPv6 */
823 ((struct sockaddr_in6 *)&from)->sin6_family = AF_INET6;
824 memcpy(&((struct sockaddr_in6 *)&from)->sin6_addr,
825 hdr.v2.addr.ip6.src_addr, 16);
826 ((struct sockaddr_in6 *)&from)->sin6_port =
827 hdr.v2.addr.ip6.src_port;
828 ((struct sockaddr_in6 *)&to)->sin6_family = AF_INET6;
829 memcpy(&((struct sockaddr_in6 *)&to)->sin6_addr,
830 hdr.v2.addr.ip6.dst_addr, 16);
831 ((struct sockaddr_in6 *)&to)->sin6_port =
832 hdr.v2.addr.ip6.dst_port;
833 goto done;
834 }
835 /* unsupported protocol, keep local connection address */
836 break;
837 case 0x00: /* LOCAL command */
838 /* keep local connection address for LOCAL */
839 break;
840 default:
841 return -1; /* not a supported command */
842 }
843 }
844 else if (ret >= 8 && memcmp(hdr.v1.line, "PROXY", 5) == 0) {
845 char *end = memchr(hdr.v1.line, '\r', ret - 1);
846 if (!end || end[1] != '\n')
847 return -1; /* partial or invalid header */
848 *end = '\0'; /* terminate the string to ease parsing */
849 size = end + 2 - hdr.v1.line; /* skip header + CRLF */
850 /* parse the V1 header using favorite address parsers like inet_pton.
851 * return -1 upon error, or simply fall through to accept.
852 */
853 }
854 else {
855 /* Wrong protocol */
856 return -1;
857 }
858
859 done:
860 /* we need to consume the appropriate amount of data from the socket */
861 do {
862 ret = recv(fd, &hdr, size, 0);
863 } while (ret == -1 && errno == EINTR);
864 return (ret >= 0) ? 1 : -1;
865 }