| Willy Tarreau | 9d752e9 | 2018-07-13 18:49:41 +0200 | [diff] [blame] | 1 | 2018-07-13 - HAProxy Internal Buffer API |
| 2 | |
| 3 | |
| 4 | 1. Background |
| 5 | |
| 6 | HAProxy uses a "struct buffer" internally to store data received from external |
| 7 | agents, as well as data to be sent to external agents. These buffers are also |
| 8 | used during data transformation such as compression, header insertion or |
| 9 | defragmentation, and are used to carry intermediary representations between the |
| 10 | various internal layers. They support wrapping at the end, and they carry their |
| 11 | own size information so that in theory it would be possible to use different |
| 12 | buffer sizes in parallel eventhough this is not currently implemented. |
| 13 | |
| 14 | The format of this structure has evolved over time, to reach a point where it |
| 15 | is convenient and versatile enough to have permitted to make several internal |
| 16 | types converge into a single one (specifically the struct chunk disappeared). |
| 17 | |
| 18 | |
| 19 | 2. Representation as of 1.9-dev1 |
| 20 | |
| 21 | The current buffer representation consists in a linear storage area of known |
| 22 | size, with a head position indicating the oldest data, and a total data count |
| 23 | expressed in bytes. The head position, data count and size are expressed as |
| 24 | integers and are positive or null. By convention, the head position is strictly |
| 25 | smaller than the buffer size and the data count is smaller than or equal to the |
| 26 | size, so that wrapping can be resolved with a single subtract. A buffer not |
| 27 | respecting these rules is said to be degenerate. Unless specified otherwise, |
| 28 | the various API functions will adopt an undefined behaviour when passed such a |
| 29 | degenerate buffer. |
| 30 | |
| 31 | Buffer declaration : |
| 32 | |
| 33 | struct buffer { |
| 34 | size_t size; // size of the storage area (wrapping point) |
| 35 | char *area; // start of the storage area |
| 36 | size_t data; // contents length after head |
| 37 | size_t head; // start offset of remaining data relative to area |
| 38 | }; |
| 39 | |
| 40 | |
| 41 | Linear buffer representation : |
| 42 | |
| 43 | area |
| 44 | | |
| 45 | V<--------------------------------------------------------->| size |
| 46 | +-----------+---------------------------------+-------------+ |
| 47 | | |/////////////////////////////////| | |
| 48 | +-----------+---------------------------------+-------------+ |
| 49 | |<--------->|<------------------------------->| |
| 50 | head data ^ |
| 51 | | |
| 52 | tail |
| 53 | |
| 54 | |
| 55 | Wrapping buffer representation : |
| 56 | |
| 57 | area |
| 58 | | |
| 59 | V<--------------------------------------------------------->| size |
| 60 | +---------------+------------------------+------------------+ |
| 61 | |///////////////| |//////////////////| |
| 62 | +---------------+------------------------+------------------+ |
| 63 | |<-------------------------------------->| head |
| 64 | |-------------->| ...data data...|<-----------------| |
| 65 | ^ |
| 66 | | |
| 67 | tail |
| 68 | |
| 69 | |
| 70 | 3. Terminology |
| 71 | |
| 72 | Manipulating a buffer just based on a head and a wrapping data count is not |
| 73 | very convenient, so we define a certain number of terms for important elements |
| 74 | characterizing a buffer : |
| 75 | |
| 76 | - origin : pointer to relative position 0 in the storage area. Undefined |
| 77 | when the buffer is not allocated. |
| 78 | |
| 79 | - size : the allocated size of the storage area starting at the origin, |
| 80 | expressed in bytes. A buffer whose size is zero is said not to |
| 81 | be allocated, and its origin in this case is undefined. |
| 82 | |
| 83 | - data : the amount of data the buffer contains, in bytes. It is always |
| 84 | lower than or equal to the buffer's size, hence it is always 0 |
| 85 | for an unallocated buffer. |
| 86 | |
| 87 | - emptiness : a buffer is said to be empty when it contains no data, hence |
| 88 | data == 0. It is possible for such buffers not to be allocated |
| 89 | and to have size == 0 as well. |
| 90 | |
| 91 | - room : the available space in the buffer. This is its size minus data. |
| 92 | |
| 93 | - head : position relative to origin where the oldest data byte is found |
| 94 | (it typically is what send() uses to pick outgoing data). The |
| 95 | head is strictly smaller than the size. |
| 96 | |
| 97 | - tail : position relative to origin where the first spare byte is found |
| 98 | (it typically is what recv() uses to store incoming data). It |
| 99 | is always equal to the buffer's data added to its head modulo |
| 100 | the buffer's size. |
| 101 | |
| 102 | - wrapping : the byte following the last one of the storage area loops back |
| 103 | to position 0. This is called wrapping. The wrapping point is |
| 104 | the first position relative to origin which doesn't belong to |
| 105 | the storage area. There is no wrapping when a buffer is not |
| 106 | allocated. Wrapping requires special care and means that the |
| 107 | regular string manipulation functions are not usable on most |
| 108 | buffers, unless it is known that no wrapping happens. Free |
| 109 | space may wrap as well if the buffer only contains data in the |
| 110 | middle. |
| 111 | |
| 112 | - alignment : a buffer is said to be aligned if its data do not wrap. That |
| 113 | is, its head is strictly before the tail, or the buffer is |
| 114 | empty and the head is null. Aligning a buffer may be required |
| 115 | to use regular string manipulation functions which have no |
| 116 | support for wrapping. |
| 117 | |
| 118 | |
| 119 | A buffer may be in three different states : |
| 120 | - unallocated : size == 0, area == 0 (b_is_null() is true) |
| 121 | - waiting : size == 0, area != 0 |
| 122 | - allocated : size > 0, area > 0 |
| 123 | |
| 124 | It is not permitted to have area == 0 with a non-null size. In addition, the |
| 125 | waiting state may also be used to indicate a read-only buffer which does not |
| 126 | wrap and which must not be freed (e.g. for use with error messages). |
| 127 | |
| 128 | The basic API only covers allocated buffers. Switching to/from the other states |
| 129 | is covered by the management API since it requires specific allocation and free |
| 130 | calls. |
| 131 | |
| 132 | |
| 133 | 4. Using buffers |
| 134 | |
| 135 | Buffers are defined in a few files : |
| 136 | - include/common/buf.h : structure definition, and manipulation functions |
| 137 | - include/common/buffer.h : resource management (alloc/free/wait lists) |
| 138 | - include/common/istbuf.h : advanced string manipulation |
| 139 | |
| 140 | |
| 141 | 4.1. Basic API |
| 142 | |
| 143 | The basic API is made of the functions which abstract accesses to the buffers |
| 144 | and which help calculating their state, free space or used space. |
| 145 | |
| 146 | ====================+==================+======================================= |
| 147 | Function | Arguments/Return | Description |
| 148 | --------------------+------------------+--------------------------------------- |
| 149 | b_is_null() | const buffer *buf| returns true if (and only if) the |
| 150 | | ret: int | buffer is not yet allocated and thus |
| 151 | | | points to a NULL area |
| 152 | --------------------+------------------+--------------------------------------- |
| 153 | b_orig() | const buffer *buf| returns the pointer to the origin of |
| 154 | | ret: char * | the storage, which is the location of |
| 155 | | | byte at offset zero. This is mostly |
| 156 | | | used by functions which handle the |
| 157 | | | wrapping by themselves |
| 158 | --------------------+------------------+--------------------------------------- |
| 159 | b_size() | const buffer *buf| returns the size of the buffer |
| 160 | | ret: size_t | |
| 161 | --------------------+------------------+--------------------------------------- |
| 162 | b_wrap() | const buffer *buf| returns the pointer to the wrapping |
| 163 | | ret: char * | position of the buffer area, which is |
| 164 | | | by definition the first byte not part |
| 165 | | | of the buffer |
| 166 | --------------------+------------------+--------------------------------------- |
| 167 | b_data() | const buffer *buf| returns the number of bytes present in |
| 168 | | ret: size_t | the buffer |
| 169 | --------------------+------------------+--------------------------------------- |
| 170 | b_room() | const buffer *buf| returns the amount of room left in the |
| 171 | | ret: size_t | buffer |
| 172 | --------------------+------------------+--------------------------------------- |
| 173 | b_full() | const buffer *buf| returns true if the buffer is full |
| 174 | | ret: int | |
| 175 | --------------------+------------------+--------------------------------------- |
| 176 | __b_stop() | const buffer *buf| returns a pointer to the byte |
| 177 | | ret: char * | following the end of the buffer, which |
| 178 | | | may be out of the buffer if the buffer |
| 179 | | | ends on the last byte of the area. It |
| 180 | | | is the caller's responsibility to |
| 181 | | | either know that the buffer does not |
| 182 | | | wrap or to check that the result does |
| 183 | | | not wrap |
| 184 | --------------------+------------------+--------------------------------------- |
| 185 | __b_stop_ofs() | const buffer *buf| returns an origin-relative offset |
| 186 | | ret: size_t | pointing to the byte following the end |
| 187 | | | of the buffer, which may be out of the |
| 188 | | | buffer if the buffer ends on the last |
| 189 | | | byte of the area. It's the caller's |
| 190 | | | responsibility to either know that the |
| 191 | | | buffer does not wrap or to check that |
| 192 | | | the result does not wrap |
| 193 | --------------------+------------------+--------------------------------------- |
| 194 | b_stop() | const buffer *buf| returns the pointer to the byte |
| 195 | | ret: char * | following the end of the buffer, which |
| 196 | | | may be out of the buffer if the buffer |
| 197 | | | ends on the last byte of the area |
| 198 | --------------------+------------------+--------------------------------------- |
| 199 | b_stop_ofs() | const buffer *buf| returns an origin-relative offset |
| 200 | | ret: size_t | pointing to the byte following the end |
| 201 | | | of the buffer, which may be out of the |
| 202 | | | buffer if the buffer ends on the last |
| 203 | | | byte of the area |
| 204 | --------------------+------------------+--------------------------------------- |
| 205 | __b_peek() | const buffer *buf| returns a pointer to the data at |
| 206 | | size_t ofs | position <ofs> relative to the head of |
| 207 | | ret: char * | the buffer. Will typically point to |
| 208 | | | input data if called with the amount |
| 209 | | | of output data. It's the caller's |
| 210 | | | responsibility to either know that the |
| 211 | | | buffer does not wrap or to check that |
| 212 | | | the result does not wrap |
| 213 | --------------------+------------------+--------------------------------------- |
| 214 | __b_peek_ofs() | const buffer *buf| returns an origin-relative offset |
| 215 | | size_t ofs | pointing to the data at position <ofs> |
| 216 | | ret: size_t | relative to the head of the |
| 217 | | | buffer. Will typically point to input |
| 218 | | | data if called with the amount of |
| 219 | | | output data. It's the caller's |
| 220 | | | responsibility to either know that the |
| 221 | | | buffer does not wrap or to check that |
| 222 | | | the result does not wrap |
| 223 | --------------------+------------------+--------------------------------------- |
| 224 | b_peek() | const buffer *buf| returns a pointer to the data at |
| 225 | | size_t ofs | position <ofs> relative to the head of |
| 226 | | ret: char * | the buffer. Will typically point to |
| 227 | | | input data if called with the amount |
| 228 | | | of output data. If applying <ofs> to |
| 229 | | | the buffers' head results in a |
| 230 | | | position between <size> and 2*>size>-1 |
| 231 | | | included, a wrapping compensation is |
| 232 | | | applied to the result |
| 233 | --------------------+------------------+--------------------------------------- |
| 234 | b_peek_ofs() | const buffer *buf| returns an origin-relative offset |
| 235 | | size_t ofs | pointing to the data at position <ofs> |
| 236 | | ret: size_t | relative to the head of the |
| 237 | | | buffer. Will typically point to input |
| 238 | | | data if called with the amount of |
| 239 | | | output data. If applying <ofs> to the |
| 240 | | | buffers' head results in a position |
| 241 | | | between <size> and 2*>size>-1 |
| 242 | | | included, a wrapping compensation is |
| 243 | | | applied to the result |
| 244 | --------------------+------------------+--------------------------------------- |
| 245 | __b_head() | const buffer *buf| returns the pointer to the buffer's |
| 246 | | ret: char * | head, which is the location of the |
| 247 | | | next byte to be dequeued. The result |
| 248 | | | is undefined for unallocated buffers |
| 249 | --------------------+------------------+--------------------------------------- |
| 250 | __b_head_ofs() | const buffer *buf| returns an origin-relative offset |
| 251 | | ret: size_t | pointing to the buffer's head, which |
| 252 | | | is the location of the next byte to be |
| 253 | | | dequeued. The result is undefined for |
| 254 | | | unallocated buffers |
| 255 | --------------------+------------------+--------------------------------------- |
| 256 | b_head() | const buffer *buf| returns the pointer to the buffer's |
| 257 | | ret: char * | head, which is the location of the |
| 258 | | | next byte to be dequeued. The result |
| 259 | | | is undefined for unallocated |
| 260 | | | buffers. If applying <ofs> to the |
| 261 | | | buffers' head results in a position |
| 262 | | | between <size> and 2*>size>-1 |
| 263 | | | included, a wrapping compensation is |
| 264 | | | applied to the result |
| 265 | --------------------+------------------+--------------------------------------- |
| 266 | b_head_ofs() | const buffer *buf| returns an origin-relative offset |
| 267 | | ret: size_t | pointing to the buffer's head, which |
| 268 | | | is the location of the next byte to be |
| 269 | | | dequeued. The result is undefined for |
| 270 | | | unallocated buffers. If applying |
| 271 | | | <ofs> to the buffers' head results in |
| 272 | | | a position between <size> and |
| 273 | | | 2*>size>-1 included, a wrapping |
| 274 | | | compensation is applied to the result |
| 275 | --------------------+------------------+--------------------------------------- |
| 276 | __b_tail() | const buffer *buf| returns the pointer to the tail of the |
| 277 | | ret: char * | buffer, which is the location of the |
| 278 | | | first byte where it is possible to |
| 279 | | | enqueue new data. The result is |
| 280 | | | undefined for unallocated buffers |
| 281 | --------------------+------------------+--------------------------------------- |
| 282 | __b_tail_ofs() | const buffer *buf| returns an origin-relative offset |
| 283 | | ret: size_t | pointing to the tail of the buffer, |
| 284 | | | which is the location of the first |
| 285 | | | byte where it is possible to enqueue |
| 286 | | | new data. The result is undefined for |
| 287 | | | unallocated buffers |
| 288 | --------------------+------------------+--------------------------------------- |
| 289 | b_tail() | const buffer *buf| returns the pointer to the tail of the |
| 290 | | ret: char * | buffer, which is the location of the |
| 291 | | | first byte where it is possible to |
| 292 | | | enqueue new data. The result is |
| 293 | | | undefined for unallocated buffers |
| 294 | --------------------+------------------+--------------------------------------- |
| 295 | b_tail_ofs() | const buffer *buf| returns an origin-relative offset |
| 296 | | ret: size_t | pointing to the tail of the buffer, |
| 297 | | | which is the location of the first |
| 298 | | | byte where it is possible to enqueue |
| 299 | | | new data. The result is undefined for |
| 300 | | | unallocated buffers |
| 301 | --------------------+------------------+--------------------------------------- |
| 302 | b_next() | const buffer *buf| for an absolute pointer <p> pointing |
| 303 | | const char *p | to a valid location within buffer <b>, |
| 304 | | ret: char * | returns the absolute pointer to the |
| 305 | | | next byte, which usually is at (p + 1) |
| 306 | | | unless p reaches the wrapping point |
| 307 | | | and wrapping is needed |
| 308 | --------------------+------------------+--------------------------------------- |
| 309 | b_next_ofs() | const buffer *buf| for an origin-relative offset <o> |
| 310 | | size_t o | pointing to a valid location within |
| 311 | | ret: size_t | buffer <b>, returns either the |
| 312 | | | relative offset pointing to the next |
| 313 | | | byte, which usually is at (o + 1) |
| 314 | | | unless o reaches the wrapping point |
| 315 | | | and wrapping is needed |
| 316 | --------------------+------------------+--------------------------------------- |
| 317 | b_dist() | const buffer *buf| returns the distance between two |
| 318 | | const char *from | pointers, taking into account the |
| 319 | | const char *to | ability to wrap around the buffer's |
| 320 | | ret: size_t | end. The operation is not defined if |
| 321 | | | either of the pointers does not belong |
| 322 | | | to the buffer or if their distance is |
| 323 | | | greater than the buffer's size |
| 324 | --------------------+------------------+--------------------------------------- |
| 325 | b_almost_full() | const buffer *buf| returns 1 if the buffer uses at least |
| 326 | | ret: int | 3/4 of its capacity, otherwise |
| 327 | | | zero. Buffers of size zero are |
| 328 | | | considered full |
| 329 | --------------------+------------------+--------------------------------------- |
| 330 | b_space_wraps() | const buffer *buf| returns non-zero only if the buffer's |
| 331 | | ret: int | free space wraps, which means that the |
| 332 | | | buffer contains data that are not |
| 333 | | | touching at least one edge |
| 334 | --------------------+------------------+--------------------------------------- |
| 335 | b_contig_data() | const buffer *buf| returns the amount of data that can |
| 336 | | size_t start | contiguously be read at once starting |
| 337 | | ret: size_t | from a relative offset <start> (which |
| 338 | | | allows to easily pre-compute blocks |
| 339 | | | for memcpy). The start point will |
| 340 | | | typically contain the amount of past |
| 341 | | | data already returned by a previous |
| 342 | | | call to this function |
| 343 | --------------------+------------------+--------------------------------------- |
| 344 | b_contig_space() | const buffer *buf| returns the amount of bytes that can |
| 345 | | ret: size_t | be appended to the buffer at once |
| 346 | --------------------+------------------+--------------------------------------- |
| 347 | b_getblk() | const buffer *buf| gets one full block of data at once |
| 348 | | char *blk | from a buffer, starting from offset |
| 349 | | size_t len | <offset> after the buffer's head, and |
| 350 | | size_t offset | limited to no more than <len> bytes. |
| 351 | | ret: size_t | The caller is responsible for ensuring |
| 352 | | | that neither <offset> nor <offset> + |
| 353 | | | <len> exceed the total number of bytes |
| 354 | | | available in the buffer. Return zero |
| 355 | | | if not enough data was available, in |
| 356 | | | which case blk is left undefined, or |
| 357 | | | the number of bytes read which is |
| 358 | | | equal to the requested size |
| 359 | --------------------+------------------+--------------------------------------- |
| 360 | b_getblk_nc() | const buffer *buf| gets one or two blocks of data at once |
| 361 | | const char **blk1| from a buffer, starting from offset |
| 362 | | size_t *len1 | <ofs> after the beginning of its |
| 363 | | const char **blk2| output, and limited to no more than |
| 364 | | size_t *len2 | <max> bytes. The caller is responsible |
| 365 | | size_t ofs | for ensuring that neither <ofs> nor |
| 366 | | size_t max | <ofs>+<max> exceed the total number of |
| 367 | | ret: int | bytes available in the buffer. Returns |
| 368 | | | 0 if not enough data were available, |
| 369 | | | or the number of blocks filled (1 or |
| 370 | | | 2). <blk1> is always filled before |
| 371 | | | <blk2>. The unused blocks are left |
| 372 | | | undefined, and the buffer is left |
| 373 | | | unaffected. Unused buffers are left in |
| 374 | | | an undefined state |
| 375 | --------------------+------------------+--------------------------------------- |
| 376 | b_reset() | buffer *buf | resets a buffer. The size is not |
| 377 | | ret: void | touched. In practice it resets the |
| 378 | | | head and the data length |
| 379 | --------------------+------------------+--------------------------------------- |
| 380 | b_sub() | buffer *buf | decreases the buffer length by <count> |
| 381 | | size_t count | without touching the head position |
| 382 | | ret: void | (only the tail moves). this may mostly |
| 383 | | | be used to trim pending data before |
| 384 | | | reusing a buffer. The caller is |
| 385 | | | responsible for not removing more than |
| 386 | | | the available data |
| 387 | --------------------+------------------+--------------------------------------- |
| 388 | b_add() | buffer *buf | increase the buffer length by <count> |
| 389 | | size_t count | without touching the head position |
| 390 | | ret: void | (only the tail moves). This is used |
| 391 | | | when adding data at the tail of a |
| 392 | | | buffer. The caller is responsible for |
| 393 | | | not adding more than the available |
| 394 | | | room |
| 395 | --------------------+------------------+--------------------------------------- |
| 396 | b_set_data() | buffer *buf | sets the buffer's length, by adjusting |
| 397 | | size_t len | the buffer's tail only. The caller is |
| 398 | | ret: void | responsible for passing a valid length |
| 399 | --------------------+------------------+--------------------------------------- |
| 400 | b_del() | buffer *buf | deletes <del> bytes at the head of |
| 401 | | size_t del | buffer <b> and updates the head. The |
| 402 | | ret: void | caller is responsible for not removing |
| 403 | | | more than the available data. This is |
| 404 | | | used after sending data from the |
| 405 | | | buffer |
| 406 | --------------------+------------------+--------------------------------------- |
| 407 | b_realign_if_empty()| buffer *buf | realigns a buffer if it's empty, does |
| 408 | | ret: void | nothing otherwise. This is mostly used |
| 409 | | | after b_del() to make an empty |
| 410 | | | buffer's free space contiguous |
| 411 | --------------------+------------------+--------------------------------------- |
| 412 | b_slow_realign() | buffer *buf | realigns a possibly wrapping buffer so |
| 413 | | size_t output | that the part remaining to be parsed |
| 414 | | ret: void | is contiguous and starts at the |
| 415 | | | beginning of the buffer and the |
| 416 | | | already parsed output part ends at the |
| 417 | | | end of the buffer. This provides the |
| 418 | | | best conditions since it allows the |
| 419 | | | largest inputs to be processed at once |
| 420 | | | and ensures that once the output data |
| 421 | | | leaves, the whole buffer is available |
| 422 | | | at once. The number of output bytes |
| 423 | | | supposedly present at the beginning of |
| 424 | | | the buffer and which need to be moved |
| 425 | | | to the end must be passed in <output>. |
| 426 | | | It will effectively make this offset |
| 427 | | | the new wrapping point. A temporary |
| 428 | | | swap area at least as large as b->size |
| 429 | | | must be provided in <swap>. It's up |
| 430 | | | to the caller to ensure <output> is no |
| 431 | | | larger than the difference between the |
| 432 | | | whole buffer's length and its input |
| 433 | --------------------+------------------+--------------------------------------- |
| 434 | b_putchar() | buffer *buf | tries to append char <c> at the end of |
| 435 | | char c | buffer <b>. Supports wrapping. New |
| 436 | | ret: void | data are silently discarded if the |
| 437 | | | buffer is already full |
| 438 | --------------------+------------------+--------------------------------------- |
| 439 | b_putblk() | buffer *buf | tries to append block <blk> at the end |
| 440 | | const char *blk | of buffer <b>. Supports wrapping. Data |
| 441 | | size_t len | are truncated if the buffer is too |
| 442 | | ret: size_t | short or if not enough space is |
| 443 | | | available. It returns the number of |
| 444 | | | bytes really copied |
| 445 | --------------------+------------------+--------------------------------------- |
| Willy Tarreau | f48919a | 2018-12-22 19:19:50 +0100 | [diff] [blame] | 446 | b_move() | buffer *buf | moves block (src,len) left or right |
| 447 | | size_t src | by <shift> bytes, supporting wrapping |
| 448 | | size_t len | and overlapping. |
| 449 | | size_t shift | |
| 450 | --------------------+------------------+--------------------------------------- |
| Willy Tarreau | 9d752e9 | 2018-07-13 18:49:41 +0200 | [diff] [blame] | 451 | b_rep_blk() | buffer *buf | writes the block <blk> at position |
| 452 | | char *pos | <pos> which must be in buffer <b>, and |
| 453 | | char *end | moves the part between <end> and the |
| 454 | | const char *blk | buffer's tail just after the end of |
| 455 | | size_t len | the copy of <blk>. This effectively |
| 456 | | ret: int | replaces the part located between |
| 457 | | | <pos> and <end> with a copy of <blk> |
| 458 | | | of length <len>. The buffer's length |
| 459 | | | is automatically updated. This is used |
| 460 | | | to replace a block with another one |
| 461 | | | inside a buffer. The shift value |
| 462 | | | (positive or negative) is returned. If |
| 463 | | | there's no space left, the move is not |
| 464 | | | done. If <len> is null, the <blk> |
| 465 | | | pointer is allowed to be null, in |
| 466 | | | order to erase a block |
| Willy Tarreau | f148888 | 2018-07-20 16:24:39 +0200 | [diff] [blame] | 467 | --------------------+------------------+--------------------------------------- |
| 468 | b_xfer() | buffer *src | transfers at most <count> bytes from |
| 469 | | buffer *dst | buffer <src> to buffer <dst> and |
| 470 | | size_t cout | returns the number of bytes copied. |
| 471 | | ret: size_t | The bytes are removed from <src> and |
| 472 | | | added to <dst>. The caller guarantees |
| 473 | | | that <count> is <= b_room(dst) |
| Willy Tarreau | 9d752e9 | 2018-07-13 18:49:41 +0200 | [diff] [blame] | 474 | ====================+==================+======================================= |
| 475 | |
| 476 | |
| 477 | 4.2. String API |
| 478 | |
| 479 | The string API aims at providing both convenient and efficient ways to read and |
| 480 | write to/from buffers using indirect strings (ist). These strings and some |
| 481 | associated functions are defined in ist.h. |
| 482 | |
| 483 | ====================+==================+======================================= |
| 484 | Function | Arguments/Return | Description |
| 485 | --------------------+------------------+--------------------------------------- |
| 486 | b_isteq() | const buffer *b | b_isteq() : returns > 0 if the first |
| 487 | | size_t o | <n> characters of buffer <b> starting |
| 488 | | size_t n | at offset <o> relative to the buffer's |
| 489 | | const ist ist | head match <ist>. (empty strings do |
| 490 | | ret: int | match). It is designed to be used with |
| 491 | | | reasonably small strings (it matches a |
| 492 | | | single byte per loop iteration). It is |
| 493 | | | expected to be used with an offset to |
| 494 | | | skip old data. Return value number of |
| 495 | | | matching bytes if >0, not enough bytes |
| 496 | | | or empty string if 0, or non-matching |
| 497 | | | byte found if <0. |
| 498 | --------------------+------------------+--------------------------------------- |
| 499 | b_isteat | struct buffer *b | b_isteat() : "eats" string <ist> from |
| 500 | | const ist ist | the head of buffer <b>. Wrapping data |
| 501 | | ret: ssize_t | is explicitly supported. It matches a |
| 502 | | | single byte per iteration so strings |
| 503 | | | should remain reasonably small. |
| 504 | | | Returns the number of bytes matched |
| 505 | | | and eaten if >0, not enough bytes or |
| 506 | | | matched empty string if 0, or non |
| 507 | | | matching byte found if <0. |
| 508 | --------------------+------------------+--------------------------------------- |
| 509 | b_istput | struct buffer *b | b_istput() : injects string <ist> at |
| 510 | | const ist ist | the tail of output buffer <b> provided |
| 511 | | ret: ssize_t | that it fits. Wrapping is supported. |
| 512 | | | It's designed for small strings as it |
| 513 | | | only writes a single byte per |
| 514 | | | iteration. Returns the number of |
| 515 | | | characters copied (ist.len), 0 if it |
| 516 | | | temporarily does not fit, or -1 if it |
| 517 | | | will never fit. It will only modify |
| 518 | | | the buffer upon success. In all cases, |
| 519 | | | the contents are copied prior to |
| 520 | | | reporting an error, so that the |
| 521 | | | destination at least contains a valid |
| 522 | | | but truncated string. |
| 523 | --------------------+------------------+--------------------------------------- |
| 524 | b_putist | struct buffer *b | b_putist() : tries to copy as much as |
| 525 | | const ist ist | possible of string <ist> into buffer |
| 526 | | ret: size_t | <b> and returns the number of bytes |
| 527 | | | copied (truncation is possible). It |
| 528 | | | uses b_putblk() and is suitable for |
| 529 | | | large blocks. |
| 530 | ====================+==================+======================================= |
| 531 | |
| 532 | |
| 533 | 4.3. Management API |
| 534 | |
| 535 | The management API makes a distinction between an empty buffer, which by |
| 536 | definition is not allocated but is ready to be allocated at any time, and a |
| 537 | buffer which failed an allocation and is waiting for an available area to be |
| 538 | offered. The functions allow to register on a list to be notified about buffer |
| 539 | availability, to notify others of a number of buffers just released, and to be |
| 540 | and to be notified of buffer availability. All allocations are made through the |
| 541 | standard buffer pools. |
| 542 | |
| 543 | ====================+==================+======================================= |
| 544 | Function | Arguments/Return | Description |
| 545 | --------------------+------------------+--------------------------------------- |
| 546 | buffer_almost_full | const buffer *buf| returns true if the buffer is not null |
| 547 | | ret: int | and at least 3/4 of the buffer's space |
| 548 | | | are used. A waiting buffer will match. |
| 549 | --------------------+------------------+--------------------------------------- |
| 550 | b_alloc | buffer *buf | allocates a buffer and assigns it to |
| 551 | | ret: buffer * | *buf. If no memory is available, (1) |
| 552 | | | is assigned instead with a zero size. |
| 553 | | | No control is made to check if *buf |
| 554 | | | already pointed to another buffer. The |
| 555 | | | allocated buffer is returned, or NULL |
| 556 | | | in case no memory is available |
| 557 | --------------------+------------------+--------------------------------------- |
| 558 | b_alloc_fast | buffer *buf | allocates a buffer and assigns it to |
| 559 | | ret: buffer * | *buf. If no memory is available, (1) |
| 560 | | | is assigned instead with a zero size. |
| 561 | | | No control is made to check if *buf |
| 562 | | | already pointed to another buffer. The |
| 563 | | | allocated buffer is returned, or NULL |
| 564 | | | in case no memory is available. The |
| 565 | | | difference with b_alloc() is that this |
| 566 | | | function only picks from the pool and |
| 567 | | | never calls malloc(), so it can fail |
| 568 | | | even if some memory is available |
| 569 | --------------------+------------------+--------------------------------------- |
| Willy Tarreau | e0d0b40 | 2019-08-08 08:06:27 +0200 | [diff] [blame] | 570 | __b_free | buffer *buf | releases <buf> which must be allocated |
| Willy Tarreau | 3b091f8 | 2019-08-08 07:53:20 +0200 | [diff] [blame] | 571 | | ret: void | and marks it empty |
| Willy Tarreau | 9d752e9 | 2018-07-13 18:49:41 +0200 | [diff] [blame] | 572 | --------------------+------------------+--------------------------------------- |
| 573 | b_free | buffer *buf | releases <buf> only if it is allocated |
| 574 | | ret: void | and marks it empty |
| 575 | --------------------+------------------+--------------------------------------- |
| 576 | b_alloc_margin | buffer *buf | ensures that <buf> is allocated. If an |
| 577 | | int margin | allocation is needed, it ensures that |
| 578 | | ret: buffer * | there are still at least <margin> |
| 579 | | | buffers available in the pool after |
| 580 | | | this allocation so that we don't leave |
| 581 | | | the pool in a condition where a |
| 582 | | | session or a response buffer could not |
| 583 | | | be allocated anymore, resulting in a |
| 584 | | | deadlock. This means that we sometimes |
| 585 | | | need to try to allocate extra entries |
| 586 | | | even if only one buffer is needed |
| 587 | --------------------+------------------+--------------------------------------- |
| 588 | offer_buffers() | void *from | offer a buffer currently belonging to |
| 589 | | uint threshold | target <from> to whoever needs |
| 590 | | ret: void | one. Any pointer is valid for <from>, |
| 591 | | | including NULL. Its purpose is to |
| 592 | | | avoid passing a buffer to oneself in |
| 593 | | | case of failed allocations (e.g. need |
| 594 | | | two buffers, get one, fail, release it |
| 595 | | | and wake up self again). In case of |
| 596 | | | normal buffer release where it is |
| 597 | | | expected that the caller is not |
| 598 | | | waiting for a buffer, NULL is fine |
| 599 | ====================+==================+======================================= |
| 600 | |
| 601 | |
| 602 | 5. Porting code from older versions |
| 603 | |
| 604 | The previous buffer API introduced in 1.5-dev9 (May 2012) used to look like the |
| 605 | following (with the struct renamed to old_buffer here to avoid confusion during |
| 606 | quick lookups at the doc). It's worth noting that the "data" field used to be |
| 607 | part of the struct but with a different type and meaning. It's important to be |
| 608 | careful about potential code making use of &b->data as it will silently compile |
| 609 | but fail. |
| 610 | |
| 611 | Previous buffer declaration : |
| 612 | |
| 613 | struct old_buffer { |
| 614 | char *p; /* buffer's start pointer, separates in and out data */ |
| 615 | unsigned int size; /* buffer size in bytes */ |
| 616 | unsigned int i; /* number of input bytes pending for analysis in the buffer */ |
| 617 | unsigned int o; /* number of out bytes the sender can consume from this buffer */ |
| 618 | char data[0]; /* <size> bytes */ |
| 619 | }; |
| 620 | |
| 621 | Previous linear buffer representation : |
| 622 | |
| 623 | data p |
| 624 | | | |
| 625 | V V |
| 626 | +-----------+--------------------+------------+-------------+ |
| 627 | | |////////////////////|////////////| | |
| 628 | +-----------+--------------------+------------+-------------+ |
| 629 | <---------------------------------------------------------> size |
| 630 | <------------------> <----------> |
| 631 | o i |
| 632 | |
| 633 | There is this correspondance between old and new fields (some will involve a |
| 634 | knowledge of a channel when the output byte count is required) : |
| 635 | |
| 636 | Old | New |
| 637 | --------+---------------------------------------------------- |
| 638 | p | data + head + co_data(channel) // ci_head(channel) |
| 639 | size | size |
| 640 | i | data - co_data(channel) // ci_data(channel) |
| 641 | o | co_data(channel) // channel->output |
| 642 | data | area |
| 643 | --------+----------------------------------------------------- |
| 644 | |
| 645 | Then some common expressions can be mapped like this : |
| 646 | |
| 647 | Old | New |
| 648 | -----------------------+--------------------------------------- |
| 649 | b->data | b_orig(b) |
| 650 | &b->data | b_orig(b) |
| 651 | bi_ptr(b) | ci_head(channel) |
| 652 | bi_end(b) | b_tail(b) |
| 653 | bo_ptr(b) | b_head(b) |
| 654 | bo_end(b) | co_tail(channel) |
| 655 | bi_putblk(b,s,l) | b_putblk(b,s,l) |
| 656 | bo_getblk(b,s,l,o) | b_getblk(b,s,l,o) |
| 657 | bo_getblk_nc(b,s,l,o) | b_getblk_nc(b,s,l,o,0,co_data(channel)) |
| 658 | b->i + b->o | b_data(b) |
| 659 | b->data + b->size | b_wrap(b) |
| 660 | b->i += len | b_add(b, len) |
| 661 | b->i -= len | b_sub(b, len) |
| 662 | b->i = len | b_set_data(b, co_data(channel) + len) |
| 663 | b->o += len | b_add(b, len); channel->output += len |
| 664 | b->o -= len | b_del(b, len); channel->output -= len |
| 665 | -----------------------+--------------------------------------- |
| 666 | |
| 667 | The buffer modification functions are less straightforward and depend a lot on |
| 668 | the context where they are used. It is strongly advised to figure in the list |
| 669 | of functions above what is available based on what is attempted to be done in |
| 670 | the existing code. |
| 671 | |
| 672 | Note that it is very likely that any out-of-tree code relying on buffers will |
| 673 | not use both ->i and ->o but instead will use exclusively ->i on the side |
| 674 | producing data and use exclusively ->o on the side consuming data (such as in a |
| 675 | mux or in an applet). In both cases, it should be assumed that the other side |
| 676 | is always zero and that either ->i or ->o is replaced with ->data, making the |
| 677 | remaining code much simpler (no more code duplication based on the data |
| 678 | direction). |