blob: 30d8737206eb5934489a7dfa0cac18eba67f2ecf [file] [log] [blame]
Willy Tarreau64a18532019-11-20 16:45:15 +01001Initialization stages aka how to get your code initialized at the right moment
2
3
41. Background
5
6Originally all subsystems were initialized via a dedicated function call
7from the huge main() function. Then some code started to become conditional
8or a bit more modular and the #ifdef placed there became a mess, resulting
9in init code being moved to function constructors in each subsystem's own
10file. Then pools of various things were introduced, starting to make the
11whole init sequence more complicated due to some forms of internal
12dependencies. Later epoll was introduced, requiring a post-fork callback,
13and finally threads arrived also requiring some post-thread init/deinit
14and allocation, marking the old architecture's last breath. Finally the
15whole thing resulted in lots of init code duplication and was simplified
16in 1.9 with the introduction of initcalls and initialization stages.
17
18
192. New architecture
20
21The new architecture relies on two layers :
22 - the registration functions
23 - the INITCALL macros and initialization stages
24
25The first ones are mostly used to add a callback to a list. The second ones
26are used to specify when to call a function. Both are totally independent,
27however they are generally combined via another set consisting in the REGISTER
28macros which make some registration functions be called at some specific points
29during the init sequence.
30
31
323. Registration functions
33
34Registration functions never fail. Or more precisely, if they fail it will only
35be on out-of-memory condition, and they will cause the process to immediately
36exit. As such they do not return any status and the caller doesn't have to care
37about their success.
38
39All available functions are described below in alphanumeric ordering. Please
40make sure to respect this ordering when adding new ones.
41
42- void hap_register_build_opts(const char *str, int must_free)
43
44 This appends the zero-terminated constant string <str> to the list of known
45 build options that will be reported on the output of "haproxy -vv". A line
46 feed character ('\n') will automatically be appended after the string when it
47 is displayed. The <must_free> argument must be zero, unless the string was
48 allocated by any malloc-compatible function such as malloc()/calloc()/
49 realloc()/strdup() or memprintf(), in which case it's better to pass a
50 non-null value so that the string is freed upon exit. Note that despite the
51 function's prototype taking a "const char *", the pointer will actually be
52 cast and freed. The const char* is here to leave more freedom to use consts
53 when making such options lists.
54
55- void hap_register_per_thread_alloc(int (*fct)())
56
57 This adds a call to function <fct> to the list of functions to be called when
58 threads are started, at the beginning of the polling loop. This is also valid
59 for the main thread and will be called even if threads are disabled, so that
60 it is guaranteed that this function will be called in any circumstance. Each
61 thread will first call all these functions exactly once when it starts. Calls
62 are serialized by the init_mutex, so that locking is not necessary in these
63 functions. There is no relation between the thread numbers and the callback
64 ordering. The function is expected to return non-zero on success, or zero on
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +050065 failure. A failure will make the process emit a succinct error message and
Willy Tarreau64a18532019-11-20 16:45:15 +010066 immediately exit. See also hap_register_per_thread_free() for functions
67 called after these ones.
68
69- void hap_register_per_thread_deinit(void (*fct)());
70
71 This adds a call to function <fct> to the list of functions to be called when
72 threads are gracefully stopped, at the end of the polling loop. This is also
73 valid for the main thread and will be called even if threads are disabled, so
74 that it is guaranteed that this function will be called in any circumstance
75 if the process experiences a soft stop. Each thread will call this function
76 exactly once when it stops. However contrary to _alloc() and _init(), the
77 calls are made without any protection, thus if any shared resource if touched
78 by the function, the function is responsible for protecting it. The reason
79 behind this is that such resources are very likely to be still in use in one
80 other thread and that most of the time the functions will in fact only touch
81 a refcount or deinitialize their private resources. See also
82 hap_register_per_thread_free() for functions called after these ones.
83
84- void hap_register_per_thread_free(void (*fct)());
85
86 This adds a call to function <fct> to the list of functions to be called when
87 threads are gracefully stopped, at the end of the polling loop, after all calls
88 to _deinit() callbacks are done for this thread. This is also valid for the
89 main thread and will be called even if threads are disabled, so that it is
90 guaranteed that this function will be called in any circumstance if the
91 process experiences a soft stop. Each thread will call this function exactly
92 once when it stops. However contrary to _alloc() and _init(), the calls are
93 made without any protection, thus if any shared resource if touched by the
94 function, the function is responsible for protecting it. The reason behind
95 this is that such resources are very likely to be still in use in one other
96 thread and that most of the time the functions will in fact only touch a
97 refcount or deinitialize their private resources. See also
98 hap_register_per_thread_deinit() for functions called before these ones.
99
100- void hap_register_per_thread_init(int (*fct)())
101
102 This adds a call to function <fct> to the list of functions to be called when
103 threads are started, at the beginning of the polling loop, right after the
104 list of _alloc() functions. This is also valid for the main thread and will
105 be called even if threads are disabled, so that it is guaranteed that this
106 function will be called in any circumstance. Each thread will call this
107 function exactly once when it starts, and calls are serialized by the
108 init_mutex which is held over all _alloc() and _init() calls, so that locking
109 is not necessary in these functions. In other words for all threads but the
110 current one, the sequence of _alloc() and _init() calls will be atomic. There
111 is no relation between the thread numbers and the callback ordering. The
112 function is expected to return non-zero on success, or zero on failure. A
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +0500113 failure will make the process emit a succinct error message and immediately
Willy Tarreau64a18532019-11-20 16:45:15 +0100114 exit. See also hap_register_per_thread_alloc() for functions called before
115 these ones.
116
William Lallemandb53eb872022-04-21 18:02:53 +0200117- void hap_register_pre_check(int (*fct)())
118
119 This adds a call to function <fct> to the list of functions to be called at
120 the step just before the configuration validity checks. This is useful when you
121 need to create things like it would have been done during the configuration
122 parsing and where the initialization should continue in the configuration
123 check.
124 It could be used for example to generate a proxy with multiple servers using
125 the configuration parser itself. At this step the trash buffers are allocated.
126 Threads are not yet started so no protection is required. The function is
127 expected to return non-zero on success, or zero on failure. A failure will make
128 the process emit a succinct error message and immediately exit.
129
Willy Tarreau64a18532019-11-20 16:45:15 +0100130- void hap_register_post_check(int (*fct)())
131
132 This adds a call to function <fct> to the list of functions to be called at
133 the end of the configuration validity checks, just at the point where the
134 program either forks or exits depending whether it's called with "-c" or not.
135 Such calls are suited for memory allocation or internal table pre-computation
136 that would preferably not be done on the fly to avoid inducing extra time to
137 a pure configuration check. Threads are not yet started so no protection is
138 required. The function is expected to return non-zero on success, or zero on
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +0500139 failure. A failure will make the process emit a succinct error message and
Willy Tarreau64a18532019-11-20 16:45:15 +0100140 immediately exit.
141
142- void hap_register_post_deinit(void (*fct)())
143
144 This adds a call to function <fct> to the list of functions to be called when
145 freeing the global sections at the end of deinit(), after everything is
146 stopped. The process is single-threaded at this point, thus these functions
147 are suitable for releasing configuration elements provided that no other
148 _deinit() function uses them, i.e. only close/release what is strictly
149 private to the subsystem. Since such functions are mostly only called during
150 soft stops (reloads) or failed startups, they tend to experience much less
151 test coverage than others despite being more exposed, and as such a lot of
152 care must be taken to test them especially when facing partial subsystem
153 initializations followed by errors.
154
155- void hap_register_post_proxy_check(int (*fct)(struct proxy *))
156
157 This adds a call to function <fct> to the list of functions to be called for
158 each proxy, after the calls to _post_server_check(). This can allow, for
159 example, to pre-configure default values for an option in a frontend based on
160 the "bind" lines or something in a backend based on the "server" lines. It's
161 worth being aware that such a function must be careful not to waste too much
162 time in order not to significantly slow down configurations with tens of
163 thousands of backends. The function is expected to return non-zero on
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +0500164 success, or zero on failure. A failure will make the process emit a succinct
Willy Tarreau64a18532019-11-20 16:45:15 +0100165 error message and immediately exit.
166
167- void hap_register_post_server_check(int (*fct)(struct server *))
168
169 This adds a call to function <fct> to the list of functions to be called for
170 each server, after the call to check_config_validity(). This can allow, for
171 example, to preset a health state on a server or to allocate a protocol-
172 specific memory area. It's worth being aware that such a function must be
173 careful not to waste too much time in order not to significantly slow down
174 configurations with tens of thousands of servers. The function is expected
175 to return non-zero on success, or zero on failure. A failure will make the
Ilya Shipitsin1fae8db2020-03-14 17:47:28 +0500176 process emit a succinct error message and immediately exit.
Willy Tarreau64a18532019-11-20 16:45:15 +0100177
178- void hap_register_proxy_deinit(void (*fct)(struct proxy *))
179
180 This adds a call to function <fct> to the list of functions to be called when
181 freeing the resources during deinit(). These functions will be called as part
182 of the proxy's resource cleanup. Note that some of the proxy's fields will
183 already have been freed and others not, so such a function must not use any
184 information from the proxy that is subject to being released. In particular,
185 all servers have already been deleted. Since such functions are mostly only
186 called during soft stops (reloads) or failed startups, they tend to
187 experience much less test coverage than others despite being more exposed,
188 and as such a lot of care must be taken to test them especially when facing
189 partial subsystem initializations followed by errors. It's worth mentioning
190 that too slow functions could have a significant impact on the configuration
191 check or exit time especially on large configurations.
192
193- void hap_register_server_deinit(void (*fct)(struct server *))
194
195 This adds a call to function <fct> to the list of functions to be called when
196 freeing the resources during deinit(). These functions will be called as part
197 of the server's resource cleanup. Note that some of the server's fields will
198 already have been freed and others not, so such a function must not use any
199 information from the server that is subject to being released. Since such
200 functions are mostly only called during soft stops (reloads) or failed
201 startups, they tend to experience much less test coverage than others despite
202 being more exposed, and as such a lot of care must be taken to test them
203 especially when facing partial subsystem initializations followed by errors.
204 It's worth mentioning that too slow functions could have a significant impact
205 on the configuration check or exit time especially on large configurations.
206
207
2084. Initialization stages
209
210In order to offer some guarantees, the startup of the program is split into
211several stages. Some callbacks can be placed into each of these stages using
212an INITCALL macro, with 0 to 3 arguments, respectively called INITCALL0 to
213INITCALL3. These macros must be placed anywhere at the top level of a C file,
214preferably at the end so that the referenced symbols have already been met,
215but it may also be fine to place them right after the callbacks themselves.
216
217Such callbacks are referenced into small structures containing a pointer to the
218function and 3 arguments. NULL replaces unused arguments. The callbacks are
219cast to (void (*)(void *, void *, void *)) and the arguments to (void *).
220
221The first argument to the INITCALL macro is the initialization stage. The
222second one is the callback function, and others if any are the arguments.
223The init stage must be among the values of the "init_stage" enum, currently,
224and in this execution order:
225
226 - STG_PREPARE : used to preset variables, pre-initialize lookup tables and
227 pre-initialize list heads
228 - STG_LOCK : used to pre-initialize locks
Willy Tarreau3ebe4d92022-02-18 14:51:49 +0100229 - STG_REGISTER : used to register static lists such as keywords
Willy Tarreau64a18532019-11-20 16:45:15 +0100230 - STG_ALLOC : used to allocate the required structures
231 - STG_POOL : used to create pools
Willy Tarreau64a18532019-11-20 16:45:15 +0100232 - STG_INIT : used to initialize subsystems
233
234Each stage is guaranteed that previous stages have successfully completed. This
Willy Tarreau3ebe4d92022-02-18 14:51:49 +0100235means that an INITCALL placed at stage STG_INIT is guaranteed that all pools
236were already created and will be usable. Conversely, an INITCALL placed at
237stage STG_REGISTER must not rely on any field that requires preliminary
Willy Tarreau64a18532019-11-20 16:45:15 +0100238allocation nor initialization. A callback cannot rely on other callbacks of the
239same stage, as the execution order within a stage is undefined and essentially
240depends on the linking order.
241
Willy Tarreau3ebe4d92022-02-18 14:51:49 +0100242The STG_REGISTER level is made for run-time linking of the various modules that
243compose the executable. Keywords, protocols and various other elements that are
244local known to each compilation unit can will be appended into common lists at
245boot time. This is why this call is placed just before STG_ALLOC.
246
Willy Tarreau64a18532019-11-20 16:45:15 +0100247Example: register a very early call to init_log() with no argument, and another
248 call to cli_register_kw(&cli_kws) much later:
249
250 INITCALL0(STG_PREPARE, init_log);
251 INITCALL1(STG_REGISTER, cli_register_kw, &cli_kws);
252
253Technically speaking, each call to such a macro adds a distinct local symbol
254whose dynamic name involves the line number. These symbols are placed into a
255separate section and the beginning and end section pointers are provided by the
256linker. When too old a linker is used, a fallback is applied consisting in
257placing them into a linked list which is built by a constructor function for
258each initcall (this takes more room).
259
260Due to the symbols internally using the line number, it is very important not
261to place more than one INITCALL per line in the source file.
262
263It is also strongly recommended that functions and referenced arguments are
264static symbols local to the source file, unless they are global registration
265functions like in the example above with cli_register_kw(), where only the
266argument is a local keywords table.
267
268INITCALLs do not expect the callback function to return anything and as such
269do not perform any error check. As such, they are very similar to constructors
270offered by the compiler except that they are segmented in stages. It is thus
271the responsibility of the called functions to perform their own error checking
272and to exit in case of error. This may change in the future.
273
274
2755. REGISTER family of macros
276
277The association of INITCALLs and registration functions allows to perform some
278early dynamic registration of functions to be used anywhere, as well as values
279to be added to existing lists without having to manipulate list elements. For
280the sake of simplification, these combinations are available as a set of
281REGISTER macros which register calls to certain functions at the appropriate
282init stage. Such macros must be used at the top level in a file, just like
283INITCALL macros. The following macros are currently supported. Please keep them
284alphanumerically ordered:
285
286- REGISTER_BUILD_OPTS(str)
287
288 Adds the constant string <str> to the list of build options. This is done by
289 registering a call to hap_register_build_opts(str, 0) at stage STG_REGISTER.
290 The string will not be freed.
291
292- REGISTER_CONFIG_POSTPARSER(name, parser)
293
294 Adds a call to function <parser> at the end of the config parsing. The
295 function is called at the very end of check_config_validity() and may be used
296 to initialize a subsystem based on global settings for example. This is done
297 by registering a call to cfg_register_postparser(name, parser) at stage
298 STG_REGISTER.
299
300- REGISTER_CONFIG_SECTION(name, parse, post)
301
302 Registers a new config section name <name> which will be parsed by function
303 <parse> (if not null), and with an optional call to function <post> at the
304 end of the section. Function <parse> must be of type (int (*parse)(const char
305 *file, int linenum, char **args, int inv)), and returns 0 on success or an
306 error code among the ERR_* set on failure. The <post> callback takes no
307 argument and returns a similar error code. This is achieved by registering a
308 call to cfg_register_section() with the three arguments at stage
309 STG_REGISTER.
310
311- REGISTER_PER_THREAD_ALLOC(fct)
312
313 Registers a call to register_per_thread_alloc(fct) at stage STG_REGISTER.
314
315- REGISTER_PER_THREAD_DEINIT(fct)
316
317 Registers a call to register_per_thread_deinit(fct) at stage STG_REGISTER.
318
319- REGISTER_PER_THREAD_FREE(fct)
320
321 Registers a call to register_per_thread_free(fct) at stage STG_REGISTER.
322
323- REGISTER_PER_THREAD_INIT(fct)
324
325 Registers a call to register_per_thread_init(fct) at stage STG_REGISTER.
326
327- REGISTER_POOL(ptr, name, size)
328
329 Used internally to declare a new pool. This is made by calling function
330 create_pool_callback() with these arguments at stage STG_POOL. Do not use it
Willy Tarreaub64ef3e2022-01-11 14:48:20 +0100331 directly, use either DECLARE_POOL() or DECLARE_STATIC_POOL() instead.
Willy Tarreau64a18532019-11-20 16:45:15 +0100332
William Lallemandb53eb872022-04-21 18:02:53 +0200333- REGISTER_PRE_CHECK(fct)
334
335 Registers a call to register_pre_check(fct) at stage STG_REGISTER.
336
Willy Tarreau64a18532019-11-20 16:45:15 +0100337- REGISTER_POST_CHECK(fct)
338
339 Registers a call to register_post_check(fct) at stage STG_REGISTER.
340
341- REGISTER_POST_DEINIT(fct)
342
343 Registers a call to register_post_deinit(fct) at stage STG_REGISTER.
344
345- REGISTER_POST_PROXY_CHECK(fct)
346
347 Registers a call to register_post_proxy_check(fct) at stage STG_REGISTER.
348
349- REGISTER_POST_SERVER_CHECK(fct)
350
351 Registers a call to register_post_server_check(fct) at stage STG_REGISTER.
352
353- REGISTER_PROXY_DEINIT(fct)
354
355 Registers a call to register_proxy_deinit(fct) at stage STG_REGISTER.
356
357- REGISTER_SERVER_DEINIT(fct)
358
359 Registers a call to register_server_deinit(fct) at stage STG_REGISTER.
360