blob: b9f82f7b6c7dcab77da7e449082a9c01f05e6008 [file] [log] [blame]
Willy Tarreauf86613f2021-05-09 06:38:07 +02001 HAProxy's peers v2.0 protocol 08/18/2016
Emeric Brun9c05c482017-11-24 18:20:57 +01002
3Author: Emeric Brun ebrun@haproxy.com
4
5
6I) Encoded Integer and Bitfield.
7
8
9 0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ]
10 240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ]
11 2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ]
12 264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ]
13 33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ]
14
15
16
17
18I) Handshake
19
20Each peer try to connect to each others, and each peer is listening
21for a connect from other peers.
22
23
24Client Server
25 Hello Message
26 ------------------------>
27
28 Status Message
29 <------------------------
30
311) Hello Message
32
33Hello message is composed of 3 lines:
34
35<protocol> <version>
36<remotepeerid>
37<localpeerid> <processpid> <relativepid>
38
Willy Tarreauf86613f2021-05-09 06:38:07 +020039protocol: current value is "HAProxyS"
Emeric Brun9c05c482017-11-24 18:20:57 +010040version: current value is "2.0"
41remotepeerid: is the name of the target peer as defined in the configuration peers section.
42localpeerid: is the name of the local peer as defined on cmdline or using hostname.
43processid: is the system process id of the local process.
44relativepid: is the haproxy's relative pid (0 if nbproc == 1)
45
462) Status Message
47
48Status message is a code followed by a LF.
49
50200: Handshake succeeded
51300: Try again later
52501: Protocol error
53502: Bad version
54503: Local peer name mismatch
55504: Remote peer name mismatch
56
57
58IV) Messages
59
60Messages:
61
620 - - - - - - - 8 - - - - - - - 16
63 Message Class| Message Type
64
65if Message Type >= 128
66
670 - - - - - - - 8 - - - - - - - 16 .....
68 Message Class| Message Type | encoded data length | data
69
70Message Classes:
710: control
721: error
7310: related to stick table updates
74255: reserved
75
76
771) Control Messages Class
78
79Available message Types for class control:
800: resync request
811: resync finished
822: resync partial
833: resync confirm
84
85
86a) Resync Request Message
87
88This message is used to request a full resync from a peer
89
90b) Resync Finished Message
91
92This message is used to signal remote peer that locally known updates have been pushed, and local peer was considered up to date.
93
94c) Resync Partial Message
95
96This message is used to signal remote peer that locally known updates have been pushed, and but the local peer is not considered up to date.
97
98d) Resync Confirm Message
99
100This message is an ack for Resync Partial or Finished Messages.
101
102It's allow the remote peer to go back to "on the fly" update process.
103
104
1052) Messages Class
106
107Available message Types for this class are:
1080: protocol error
1091: size limit reached
110
111a) Protocol Message
112
Joseph Herlant71b4b152018-11-13 16:55:16 -0800113To signal that a protocol error occurred. Connection will be shutdown just after sending this message.
Emeric Brun9c05c482017-11-24 18:20:57 +0100114
115b) Size Limit Error Message
116
117To signal that a message is outsized and can not be correctly handled. Connection will be broken.
118
119
120
1213) Stick Table Updates Messages Class
122
123Available message Types for this class are:
1240: Entry update
1251: Incremental entry update
1262: table definition
1273: table switch
1284: updates ack message.
129
130
131a) Update Message
132
1330 - - - - - - - 8 - - - - - - - 16 .....
134 Message class | Message Type | encoded data length | data
135
136
137data is composed like this
138
1390 - - - - - - - 32 .............................
140Local Update ID | Key value | data values ....
141
142Update ID in a 32bits identifier of the local update.
143
144Key value format depends of the table key type:
145
146- for keytype string
147
1480 .................................
149encoded string length | string value
150
151- for keytype integer
1520 - - - - - - - - - - 32
153encoded integer value |
154
155- for other key type
156
157The value length is annonced in table definition message
1580 ....................
159value
160
161
162b) Incremental Update Message
163
164Same format than update message except the Update ID is not present, the receiver should
165consider that the update ID is an increment of 1 of the previous considered update message (partial or not)
166
167
168c) Table Definition Message
169
170This message is used by the receiver to identify the stick table concerned by next update messages and
171to know which data is pushed in these updates.
172
1730 - - - - - - - 8 - - - - - - - 16 .....
174 Message class | Message Type | encoded data length | data
175
176
177data is composed like this
178
1790 ...................................................................
180Encoded Sender Table Id | Encoded Local Table Name Length | Table Name | Encoded Table Type | Encoded Table Keylen | Encoded Table Data Types Bitfield
181
182
183Encoded Sender Table ID present a the id numerical ID affected to that table by the sender
184It will be used by "Updates Aknowlegement Messages" and "Table Switch Messages".
185
186Encoded Local Table Name Length present the length to read the table name.
187
188"Table Name" is the shared identifier of the table (name of the current table in the configuration)
189It permits the receiver to identify the concerned table. The receiver should keep in memory the matching
190between the "Sender Table ID" to identify it directly in case of "Table Switch Message".
191
192Table Type present the numeric type of key used to store stick table entries:
193integer
Emeric Brunc329cd02020-06-02 11:17:42 +0200194 2: signed integer
195 4: IPv4 address
196 5: IPv6 address
197 6: string
198 7: binary
Emeric Brun9c05c482017-11-24 18:20:57 +0100199
200Table Keylen present the key length or max length in case of strings or binary (padded with 0).
201
202Data Types Bitfield present the types of data linearly pushed in next updates message (they will be linearly pushed in the update message)
203Known types are
204bit
205 0: server id
206 1: gpt0
207 2: gpc0
208 3: gpc0 rate
209 4: connections counter
210 5: connection rate
211 6: number of current connections
212 7: sessions counter
213 8: session rate
214 9: http requests counter
215 10: http requests rate
216 11: errors counter
217 12: errors rate
218 13: bytes in counter
219 14: bytes in rate
220 15: bytes out rate
221 16: bytes out rate
Frédéric Lécaille6778b272018-01-29 15:22:53 +0100222 17: gpc1
223 18: gpc1 rate
Emeric Brun9c05c482017-11-24 18:20:57 +0100224
225d) Table Switch Message
226
227After a Table Message Define, this message can be used by the receiver to identify the stick table concerned by next update messages.
228
2290 - - - - - - - 8 - - - - - - - 16 .....
230 Message class | Message Type | encoded data length | data
231
232
233data is composed like this
234
235
2360 .....................
237encoded Sender Table Id
238
239c) Update Ack Message
240
2410 - - - - - - - 8 - - - - - - - 16 .....
242 Message class | Message Type | encoded data length | data
243
244data is composed like this
245
2460 ....................... - - - - - - - - 32
247Encoded Remote Table Id | Update Id
248
249
250Remote Table Id is the numeric identifier of the table on the remote side.
Joseph Herlant71b4b152018-11-13 16:55:16 -0800251Update Id is the id of the last update locally committed.
Emeric Brun9c05c482017-11-24 18:20:57 +0100252
Joseph Herlant71b4b152018-11-13 16:55:16 -0800253If a re-connection occurred, the sender should know he will have to restart the push of updates from this point.
Emeric Brun9c05c482017-11-24 18:20:57 +0100254
255III) Initial full resync process.
256
257
258a) Resync from local old process
259
260An old soft-stopped process will close all established sessions with remote peers and will try to connect to a new
261local process to push all known ending with a Resync Finished Message or a Resync Partial Message (if it it does not consider itself as full updated).
262
263A new process will wait for a an incoming connection from a local process during 5 seconds. It will learn the updates from this
264process until he will receive a Resync Finished Message or a Resync Partial Message. If it receive a Resync Finished Message it will consider itself
265as fully updated and stops to ask for resync. If it receive a Resync Partial Message it will wait once again for 5 seconds for an other incoming connection from a local process.
266Same thing if the session was broken before receiving any "Resync Partial Message" or "Resync Finished Message".
267
268If one of these 5 seconds timeout expire, the process will try to request resync from a remote connected peer (see b). The process will wait until 5seconds
269if no available remote peers are found.
270
271If the timeout expire, the process will consider itself ass fully updated
272
273b) Resync from remote peers
274
275The process will randomly choose a remote connected peer and ask for a full resync using a Resync Request Message. The process will wait until 5seconds
276if no available remote peers are found.
277
278The chosen remote peer will push its all known data ending with a Resync Finished Message or a Resync Partial Message (if it it does not consider itself as full updated).
279
Michael Prokop4438c602019-05-24 10:25:45 +0200280If it receives a Resync Finished Message it will consider itself as fully updated and stops to ask for resync.
Emeric Brun9c05c482017-11-24 18:20:57 +0100281
Michael Prokop4438c602019-05-24 10:25:45 +0200282If it receives a Resync Partial Message, the current peer will be flagged to anymore be requested and any other connected peer will be randomly chosen for a resync request (5s).
Emeric Brun9c05c482017-11-24 18:20:57 +0100283
Michael Prokop4438c602019-05-24 10:25:45 +0200284If the session is broken before receiving any of these messages any other connected peer will be randomly chosen for a resync request (5s).
Emeric Brun9c05c482017-11-24 18:20:57 +0100285
286If the timeout expire, the process will consider itself as fully updated
287
288