| Frédéric Lécaille | a4d0361 | 2017-06-14 15:16:15 +0200 | [diff] [blame] | 1 | ** Plug queueing disciplines ** |
| 2 | |
| 3 | The 'plug' qdisc type is not documented. It is even not supported |
| 4 | by traffic shaping tools like 'tc' from iproute2 package. |
| 5 | |
| 6 | Such qdiscs have already been used by Yelp engineers but outside |
| 7 | of haproxy with libnl-utils tools (especially nl-qdisc-* tools) |
| 8 | to implement a workaround and make haproxy reloads work. |
| 9 | |
| 10 | Indeed with such plug qdiscs coupled with iptables configurations |
| 11 | we are able to temporarily bufferize IP packets and to release them as |
| 12 | needed. So, they may be very useful to "synchronize" TCP sessions |
| 13 | or at higher level to put network applications in states approaching |
| 14 | the ones suspected to occur during bugs. Furthermore to be sure |
| 15 | to produce a correct bug fix, it may be useful to reproduce |
| 16 | as mush as needed such painful bugs. This is where plug qdiscs |
| 17 | may be useful. |
| 18 | |
| 19 | To have an idea about how to use plug qdisc on the command line I highly recommend to |
| 20 | read Willy Tarreau blog here: |
| 21 | |
| 22 | https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ |
| 23 | |
| 24 | which refers to this other one from Yelp: |
| 25 | |
| 26 | https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html |
| 27 | |
| 28 | The code found in plug_qdisc.c file already helped in fixing a painful bug hard to |
| 29 | fix because hard to reproduce. To use the API it exports this is quite easy: |
| 30 | |
| 31 | - First your program must call plug_disc_attach() to create if not already created |
| 32 | a plug qdisc and use it (must be done during your application own already existing |
| 33 | initializations). |
| 34 | Note that this function calls plug_qdisc_release_indefinite_buffer() so that to |
| 35 | release already buffered packets before you start your application, |
| 36 | |
| 37 | - then call plug_qdisc_plug_buffer() to start buffering packets incoming to your |
| 38 | plug qdisc. So they won't be delivered to your application, |
| 39 | |
| 40 | - then call plug_qdisc_release_indefinite_buffer() to stop buffering the packets |
| 41 | incoming to your plug qdisc and release those already buffered. |
| 42 | So, that to be deliver them to your application. |
| 43 | |
| 44 | This code is short and simple. But uses several libraries especially libnl-route module |
| 45 | part of libnl library. To compile haproxy and make it use the plug_qdisc.c code we had |
| 46 | to link it against several libnl3 library modules like that: |
| 47 | |
| 48 | -lnl-genl-3 -lnl-route-3 -lnl-3 -lnl-cli-3 |
| 49 | |
| 50 | |
| 51 | - Some references: |
| 52 | Libnl API documentation may be found here: |
| 53 | https://www.infradead.org/~tgr/libnl/doc/api/index.html |
| 54 | |
| 55 | Kernel sources: |
| 56 | http://elixir.free-electrons.com/linux/latest/source/net/sched/sch_plug.c |
| 57 | |
| 58 | Nice website about traffic shaping with queuing disciplines: |
| 59 | http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:networking:traffic_control |