doc/proxy-protocol.txt - haproxy - Gitiles

 The PROXY protocol - 2010/10/29 - Willy TARREAU
 -----------------------------------------------

 Relaying TCP connections through proxies generally involves a loss of the
 original TCP connection parameters such as source and destination addresses,
 ports, and so on. Some protocols make it a little bit easier to transfer such
 information. For SMTP, Postfix authors have proposed the XCLIENT protocol which
 received broad adoption and is particularly suited to mail exchanges. In HTTP,
 we have the non-standard but omnipresent X-Forwarded-For header which relays
 information about the original source address, and the less common
 X-Original-To which relays information about the destination address.

 However, both mechanisms require a knowledge of the underlying protocol to be
 implemented in intermediaries.

 Then comes a new class of products which we'll call "dumb proxies", not because
 they don't do anything, but because they're processing protocol-agnostic data.
 Stunnel is an example of such a "dumb proxy". It talks raw TCP on one side, and
 raw SSL on the other one, and does that reliably.

 The problem with such a proxy when it is combined with another one such as
 haproxy is to adapt it to talk the higher level protocol. A patch is available
 for Stunnel to make it capable to insert an X-Forwarded-For header in the first
 HTTP request of each incoming connection. Haproxy is able not to add another
 one when the connection comes from Stunnel, so that it's possible to hide it
 from the servers.

 The typical architecture becomes the following one :


       +--------+      HTTP                      :80 +----------+
       | client |  --------------------------------> |          |
       |        |                                    | haproxy, |
       +--------+             +---------+            |  1 or 2  |
      /        /     HTTPS    | stunnel |  HTTP  :81 | listening|
     <________/    ---------> | (server | ---------> |  ports   |
                              |  mode)  |            |          |
                              +---------+            +----------+


 The problem appears when haproxy runs with keep-alive on the side towards the
 client. The Stunnel patch will only add the X-Forwarded-For header to the first
 request of each connection and all subsequent requests will not have it. One
 solution could be to improve the patch to make it support keep-alive and parse
 all forwarded data, whether they're announced with a Content-Length or with a
 Transfer-Encoding, taking care of special methods such as HEAD which announce
 data without transfering them, etc... In fact, it would require implementing a
 full HTTP stack in Stunnel. It would then become a lot more complex, a lot less
 reliable and would not anymore be the "dumb proxy" that fits every purposes.

 In practice, we don't need to add a header for each request because we'll emit
 the exact same information every time : the information related to the client
 side connection. We could then cache that information in haproxy and use it for
 every other request. But that becomes dangerous and is still limited to HTTP
 only.

 Another approach would be to prepend each connection with a line reporting the
 characteristics of the other side's connection. This method is a lot simpler to
 implement, does not require any protocol-specific knowledge on either side, and
 completely fits the purpose. That's finally what we did with a small patch to
 Stunnel and another one to haproxy. We have called this protocol the PROXY
 protocol.

 The PROXY protocol's goal is to fill the receiver's internal structures with
 the information it could have found itself if it performed the accept from the
 client. Thus right now we're supporting the following :
   - INET protocol and family (TCP over IPv4 or IPv6)
   - layer 3 source and destination addresses
   - layer 4 source and destination ports if any

 Unlike the XCLIENT protocol, the PROXY protocol was designed with limited
 extensibility in order to help the receiver parse it very fast, while keeping
 it human-readable for better debugging possibilities. So it consists in exactly
 the following block prepended before any data flowing from the dumb proxy to
 the next hop :

   - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 )

   - exactly one space : " " ( \x20 )

   - a string indicating the proxied INET protocol and family. At the moment,
     only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6"
     ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Unsupported or
     unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E \x4B
     \x4E \x4F \x57 \x4E). The remaining fields of the line are then optional
     and may be ignored, until the CRLF is found.

   - exactly one space : " " ( \x20 )

   - the layer 3 source address in its canonical format. IPv4 addresses must be
     indicated as a series of exactly 4 integers in the range [0..255] inclusive
     written in decimal representation separated by exactly one dot between each
     other. Heading zeroes are not permitted in front of numbers in order to
     avoid any possible confusion with octal numbers. IPv6 addresses must be
     indicated as series of 4 hexadecimal digits (upper or lower case) delimited
     by colons between each other, with the acceptance of one double colon
     sequence to replace the largest acceptable range of consecutive zeroes. The
     total number of decoded bits must exactly be 128. The advertised protocol
     family dictates what format to use.

   - exactly one space : " " ( \x20 )

   - the layer 3 destination address in its canonical format. It is the same
     format as the layer 3 source address and matches the same family.

   - exactly one space : " " ( \x20 )

   - the TCP source port represented as a decimal integer in the range
     [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
     in order to avoid any possible confusion with octal numbers.

   - exactly one space : " " ( \x20 )

   - the TCP destination port represented as a decimal integer in the range
     [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
     in order to avoid any possible confusion with octal numbers.

   - the CRLF sequence ( \x0D \x0A )

 The receiver MUST be configured to only receive this protocol and MUST not try
 to guess whether the line is prepended or not. That means that the protocol
 explicitly prevents port sharing between public and private access. Otherwise
 it would become a big security issue. The receiver should ensure proper access
 filtering so that only trusted proxies are allowed to use this protocol. The
 receiver must wait for the CRLF sequence to decode the addresses in order to
 ensure they are complete. Any sequence which does not exactly match the
 protocol must be discarded and cause a connection abort. It is recommended
 to abort the connection as soon as possible to that the emitter notices the
 anomaly.

 If the announced transport protocol is "UNKNOWN", then the receiver knows that
 the emitter talks the correct protocol, any may or may not decide to accept the
 connection and use the real connection's parameters as if there was no such
 protocol on the wire.

 An example of such a line before an HTTP request would look like this (CR
 marked as "\r" and LF marked as "\n") :

     PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
     GET / HTTP/1.1\r\n
     Host: 192.168.0.11\r\n
     \r\n

 For the emitter, the line is easy to put into the output buffers once the
 connection is established. For the receiver, once the line is parsed, it's
 easy to skip it from the input buffers.

 We have a patch available for recent versions of Stunnel that brings it the
 ability to be an emitter. The feature is called "sendproxy" there. The code
 for the receiving side has been merged into haproxy and is enabled using the
 "accept-proxy" keyword on a "bind" statement. Haproxy will use the transport
 information from the PROXY protocol for logging, ACLs, etc... everywhere an
 information about the original connection is required.

 It is possible that the protocol may slightly evolve to present other
 information such as the incoming network interface, or the origin addresses in
 case of network address translation happening before the first proxy, but this
 is not identified as a requirement right now.
 --
	The PROXY protocol - 2010/10/29 - Willy TARREAU
	-----------------------------------------------

	Relaying TCP connections through proxies generally involves a loss of the
	original TCP connection parameters such as source and destination addresses,
	ports, and so on. Some protocols make it a little bit easier to transfer such
	information. For SMTP, Postfix authors have proposed the XCLIENT protocol which
	received broad adoption and is particularly suited to mail exchanges. In HTTP,
	we have the non-standard but omnipresent X-Forwarded-For header which relays
	information about the original source address, and the less common
	X-Original-To which relays information about the destination address.

	However, both mechanisms require a knowledge of the underlying protocol to be
	implemented in intermediaries.

	Then comes a new class of products which we'll call "dumb proxies", not because
	they don't do anything, but because they're processing protocol-agnostic data.
	Stunnel is an example of such a "dumb proxy". It talks raw TCP on one side, and
	raw SSL on the other one, and does that reliably.

	The problem with such a proxy when it is combined with another one such as
	haproxy is to adapt it to talk the higher level protocol. A patch is available
	for Stunnel to make it capable to insert an X-Forwarded-For header in the first
	HTTP request of each incoming connection. Haproxy is able not to add another
	one when the connection comes from Stunnel, so that it's possible to hide it
	from the servers.

	The typical architecture becomes the following one :


	+--------+ HTTP :80 +----------+
	\| client \| --------------------------------> \| \|
	\| \| \| haproxy, \|
	+--------+ +---------+ \| 1 or 2 \|
	/ / HTTPS \| stunnel \| HTTP :81 \| listening\|
	<________/ ---------> \| (server \| ---------> \| ports \|
	\| mode) \| \| \|
	+---------+ +----------+


	The problem appears when haproxy runs with keep-alive on the side towards the
	client. The Stunnel patch will only add the X-Forwarded-For header to the first
	request of each connection and all subsequent requests will not have it. One
	solution could be to improve the patch to make it support keep-alive and parse
	all forwarded data, whether they're announced with a Content-Length or with a
	Transfer-Encoding, taking care of special methods such as HEAD which announce
	data without transfering them, etc... In fact, it would require implementing a
	full HTTP stack in Stunnel. It would then become a lot more complex, a lot less
	reliable and would not anymore be the "dumb proxy" that fits every purposes.

	In practice, we don't need to add a header for each request because we'll emit
	the exact same information every time : the information related to the client
	side connection. We could then cache that information in haproxy and use it for
	every other request. But that becomes dangerous and is still limited to HTTP
	only.

	Another approach would be to prepend each connection with a line reporting the
	characteristics of the other side's connection. This method is a lot simpler to
	implement, does not require any protocol-specific knowledge on either side, and
	completely fits the purpose. That's finally what we did with a small patch to
	Stunnel and another one to haproxy. We have called this protocol the PROXY
	protocol.

	The PROXY protocol's goal is to fill the receiver's internal structures with
	the information it could have found itself if it performed the accept from the
	client. Thus right now we're supporting the following :
	- INET protocol and family (TCP over IPv4 or IPv6)
	- layer 3 source and destination addresses
	- layer 4 source and destination ports if any

	Unlike the XCLIENT protocol, the PROXY protocol was designed with limited
	extensibility in order to help the receiver parse it very fast, while keeping
	it human-readable for better debugging possibilities. So it consists in exactly
	the following block prepended before any data flowing from the dumb proxy to
	the next hop :

	- a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 )

	- exactly one space : " " ( \x20 )

	- a string indicating the proxied INET protocol and family. At the moment,
	only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6"
	( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Unsupported or
	unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E \x4B
	\x4E \x4F \x57 \x4E). The remaining fields of the line are then optional
	and may be ignored, until the CRLF is found.

	- exactly one space : " " ( \x20 )

	- the layer 3 source address in its canonical format. IPv4 addresses must be
	indicated as a series of exactly 4 integers in the range [0..255] inclusive
	written in decimal representation separated by exactly one dot between each
	other. Heading zeroes are not permitted in front of numbers in order to
	avoid any possible confusion with octal numbers. IPv6 addresses must be
	indicated as series of 4 hexadecimal digits (upper or lower case) delimited
	by colons between each other, with the acceptance of one double colon
	sequence to replace the largest acceptable range of consecutive zeroes. The
	total number of decoded bits must exactly be 128. The advertised protocol
	family dictates what format to use.

	- exactly one space : " " ( \x20 )

	- the layer 3 destination address in its canonical format. It is the same
	format as the layer 3 source address and matches the same family.

	- exactly one space : " " ( \x20 )

	- the TCP source port represented as a decimal integer in the range
	[0..65535] inclusive. Heading zeroes are not permitted in front of numbers
	in order to avoid any possible confusion with octal numbers.

	- exactly one space : " " ( \x20 )

	- the TCP destination port represented as a decimal integer in the range
	[0..65535] inclusive. Heading zeroes are not permitted in front of numbers
	in order to avoid any possible confusion with octal numbers.

	- the CRLF sequence ( \x0D \x0A )

	The receiver MUST be configured to only receive this protocol and MUST not try
	to guess whether the line is prepended or not. That means that the protocol
	explicitly prevents port sharing between public and private access. Otherwise
	it would become a big security issue. The receiver should ensure proper access
	filtering so that only trusted proxies are allowed to use this protocol. The
	receiver must wait for the CRLF sequence to decode the addresses in order to
	ensure they are complete. Any sequence which does not exactly match the
	protocol must be discarded and cause a connection abort. It is recommended
	to abort the connection as soon as possible to that the emitter notices the
	anomaly.

	If the announced transport protocol is "UNKNOWN", then the receiver knows that
	the emitter talks the correct protocol, any may or may not decide to accept the
	connection and use the real connection's parameters as if there was no such
	protocol on the wire.

	An example of such a line before an HTTP request would look like this (CR
	marked as "\r" and LF marked as "\n") :

	PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
	GET / HTTP/1.1\r\n
	Host: 192.168.0.11\r\n
	\r\n

	For the emitter, the line is easy to put into the output buffers once the
	connection is established. For the receiver, once the line is parsed, it's
	easy to skip it from the input buffers.

	We have a patch available for recent versions of Stunnel that brings it the
	ability to be an emitter. The feature is called "sendproxy" there. The code
	for the receiving side has been merged into haproxy and is enabled using the
	"accept-proxy" keyword on a "bind" statement. Haproxy will use the transport
	information from the PROXY protocol for logging, ACLs, etc... everywhere an
	information about the original connection is required.

	It is possible that the protocol may slightly evolve to present other
	information such as the incoming network interface, or the origin addresses in
	case of network address translation happening before the first proxy, but this
	is not identified as a requirement right now.
	--