Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 1 | /* |
| 2 | * File descriptors management functions. |
| 3 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 4 | * Copyright 2000-2014 Willy Tarreau <w@1wt.eu> |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 5 | * |
| 6 | * This program is free software; you can redistribute it and/or |
| 7 | * modify it under the terms of the GNU General Public License |
| 8 | * as published by the Free Software Foundation; either version |
| 9 | * 2 of the License, or (at your option) any later version. |
| 10 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 11 | * This code implements an events cache for file descriptors. It remembers the |
| 12 | * readiness of a file descriptor after a return from poll() and the fact that |
| 13 | * an I/O attempt failed on EAGAIN. Events in the cache which are still marked |
| 14 | * ready and active are processed just as if they were reported by poll(). |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 15 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 16 | * This serves multiple purposes. First, it significantly improves performance |
| 17 | * by avoiding to subscribe to polling unless absolutely necessary, so most |
| 18 | * events are processed without polling at all, especially send() which |
| 19 | * benefits from the socket buffers. Second, it is the only way to support |
| 20 | * edge-triggered pollers (eg: EPOLL_ET). And third, it enables I/O operations |
| 21 | * that are backed by invisible buffers. For example, SSL is able to read a |
| 22 | * whole socket buffer and not deliver it to the application buffer because |
| 23 | * it's full. Unfortunately, it won't be reported by a poller anymore until |
| 24 | * some new activity happens. The only way to call it again thus is to keep |
| 25 | * this readiness information in the cache and to access it without polling |
| 26 | * once the FD is enabled again. |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 27 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 28 | * One interesting feature of the cache is that it maintains the principle |
| 29 | * of speculative I/O introduced in haproxy 1.3 : the first time an event is |
| 30 | * enabled, the FD is considered as ready so that the I/O attempt is performed |
| 31 | * via the cache without polling. And the polling happens only when EAGAIN is |
| 32 | * first met. This avoids polling for HTTP requests, especially when the |
| 33 | * defer-accept mode is used. It also avoids polling for sending short data |
| 34 | * such as requests to servers or short responses to clients. |
| 35 | * |
| 36 | * The cache consists in a list of active events and a list of updates. |
| 37 | * Active events are events that are expected to come and that we must report |
| 38 | * to the application until it asks to stop or asks to poll. Updates are new |
| 39 | * requests for changing an FD state. Updates are the only way to create new |
| 40 | * events. This is important because it means that the number of cached events |
| 41 | * cannot increase between updates and will only grow one at a time while |
| 42 | * processing updates. All updates must always be processed, though events |
| 43 | * might be processed by small batches if required. |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 44 | * |
| 45 | * There is no direct link between the FD and the updates list. There is only a |
| 46 | * bit in the fdtab[] to indicate than a file descriptor is already present in |
| 47 | * the updates list. Once an fd is present in the updates list, it will have to |
| 48 | * be considered even if its changes are reverted in the middle or if the fd is |
| 49 | * replaced. |
| 50 | * |
| 51 | * It is important to understand that as long as all expected events are |
| 52 | * processed, they might starve the polled events, especially because polled |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 53 | * I/O starvation quickly induces more cached I/O. One solution to this |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 54 | * consists in only processing a part of the events at once, but one drawback |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 55 | * is that unhandled events will still wake the poller up. Using an edge- |
| 56 | * triggered poller such as EPOLL_ET will solve this issue though. |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 57 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 58 | * Since we do not want to scan all the FD list to find cached I/O events, |
| 59 | * we store them in a list consisting in a linear array holding only the FD |
| 60 | * indexes right now. Note that a closed FD cannot exist in the cache, because |
| 61 | * it is closed by fd_delete() which in turn calls fd_release_cache_entry() |
| 62 | * which always removes it from the list. |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 63 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 64 | * The FD array has to hold a back reference to the cache. This reference is |
| 65 | * always valid unless the FD is not in the cache and is not updated, in which |
| 66 | * case the reference points to index 0. |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 67 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 68 | * The event state for an FD, as found in fdtab[].state, is maintained for each |
| 69 | * direction. The state field is built this way, with R bits in the low nibble |
| 70 | * and W bits in the high nibble for ease of access and debugging : |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 71 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 72 | * 7 6 5 4 3 2 1 0 |
| 73 | * [ 0 | PW | RW | AW | 0 | PR | RR | AR ] |
| 74 | * |
| 75 | * A* = active *R = read |
| 76 | * P* = polled *W = write |
| 77 | * R* = ready |
| 78 | * |
| 79 | * An FD is marked "active" when there is a desire to use it. |
| 80 | * An FD is marked "polled" when it is registered in the polling. |
| 81 | * An FD is marked "ready" when it has not faced a new EAGAIN since last wake-up |
| 82 | * (it is a cache of the last EAGAIN regardless of polling changes). |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 83 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 84 | * We have 8 possible states for each direction based on these 3 flags : |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 85 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 86 | * +---+---+---+----------+---------------------------------------------+ |
| 87 | * | P | R | A | State | Description | |
| 88 | * +---+---+---+----------+---------------------------------------------+ |
| 89 | * | 0 | 0 | 0 | DISABLED | No activity desired, not ready. | |
| 90 | * | 0 | 0 | 1 | MUSTPOLL | Activity desired via polling. | |
| 91 | * | 0 | 1 | 0 | STOPPED | End of activity without polling. | |
| 92 | * | 0 | 1 | 1 | ACTIVE | Activity desired without polling. | |
| 93 | * | 1 | 0 | 0 | ABORT | Aborted poll(). Not frequently seen. | |
| 94 | * | 1 | 0 | 1 | POLLED | FD is being polled. | |
| 95 | * | 1 | 1 | 0 | PAUSED | FD was paused while ready (eg: buffer full) | |
| 96 | * | 1 | 1 | 1 | READY | FD was marked ready by poll() | |
| 97 | * +---+---+---+----------+---------------------------------------------+ |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 98 | * |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 99 | * The transitions are pretty simple : |
| 100 | * - fd_want_*() : set flag A |
| 101 | * - fd_stop_*() : clear flag A |
| 102 | * - fd_cant_*() : clear flag R (when facing EAGAIN) |
| 103 | * - fd_may_*() : set flag R (upon return from poll()) |
| 104 | * - sync() : if (A) { if (!R) P := 1 } else { P := 0 } |
| 105 | * |
| 106 | * The PAUSED, ABORT and MUSTPOLL states are transient for level-trigerred |
| 107 | * pollers and are fixed by the sync() which happens at the beginning of the |
| 108 | * poller. For event-triggered pollers, only the MUSTPOLL state will be |
| 109 | * transient and ABORT will lead to PAUSED. The ACTIVE state is the only stable |
| 110 | * one which has P != A. |
| 111 | * |
| 112 | * The READY state is a bit special as activity on the FD might be notified |
| 113 | * both by the poller or by the cache. But it is needed for some multi-layer |
| 114 | * protocols (eg: SSL) where connection activity is not 100% linked to FD |
| 115 | * activity. Also some pollers might prefer to implement it as ACTIVE if |
| 116 | * enabling/disabling the FD is cheap. The READY and ACTIVE states are the |
| 117 | * two states for which a cache entry is allocated. |
| 118 | * |
| 119 | * The state transitions look like the diagram below. Only the 4 right states |
| 120 | * have polling enabled : |
| 121 | * |
| 122 | * (POLLED=0) (POLLED=1) |
| 123 | * |
| 124 | * +----------+ sync +-------+ |
| 125 | * | DISABLED | <----- | ABORT | (READY=0, ACTIVE=0) |
| 126 | * +----------+ +-------+ |
| 127 | * clr | ^ set | ^ |
| 128 | * | | | | |
| 129 | * v | set v | clr |
| 130 | * +----------+ sync +--------+ |
| 131 | * | MUSTPOLL | -----> | POLLED | (READY=0, ACTIVE=1) |
| 132 | * +----------+ +--------+ |
| 133 | * ^ poll | ^ |
| 134 | * | | | |
| 135 | * | EAGAIN v | EAGAIN |
| 136 | * +--------+ +-------+ |
| 137 | * | ACTIVE | | READY | (READY=1, ACTIVE=1) |
| 138 | * +--------+ +-------+ |
| 139 | * clr | ^ set | ^ |
| 140 | * | | | | |
| 141 | * v | set v | clr |
| 142 | * +---------+ sync +--------+ |
| 143 | * | STOPPED | <------ | PAUSED | (READY=1, ACTIVE=0) |
| 144 | * +---------+ +--------+ |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 145 | */ |
| 146 | |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 147 | #include <stdio.h> |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 148 | #include <string.h> |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 149 | #include <unistd.h> |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 150 | #include <sys/types.h> |
| 151 | |
Willy Tarreau | 2dd0d47 | 2006-06-29 17:53:05 +0200 | [diff] [blame] | 152 | #include <common/compat.h> |
| 153 | #include <common/config.h> |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 154 | |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 155 | #include <types/global.h> |
| 156 | |
Willy Tarreau | 2a42950 | 2006-10-15 14:52:29 +0200 | [diff] [blame] | 157 | #include <proto/fd.h> |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 158 | #include <proto/log.h> |
Willy Tarreau | c6f4ce8 | 2009-06-10 11:09:37 +0200 | [diff] [blame] | 159 | #include <proto/port_range.h> |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 160 | |
| 161 | struct fdtab *fdtab = NULL; /* array of all the file descriptors */ |
Willy Tarreau | 8d5d77e | 2009-10-18 07:25:52 +0200 | [diff] [blame] | 162 | struct fdinfo *fdinfo = NULL; /* less-often used infos for file descriptors */ |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 163 | int totalconn; /* total # of terminated sessions */ |
| 164 | int actconn; /* # of active sessions */ |
| 165 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 166 | struct poller pollers[MAX_POLLERS]; |
| 167 | struct poller cur_poller; |
| 168 | int nbpollers = 0; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 169 | |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 170 | volatile struct fdlist fd_cache ; // FD events cache |
| 171 | volatile struct fdlist fd_cache_local[MAX_THREADS]; // FD events local for each thread |
| 172 | |
Christopher Faulet | 69553fe | 2018-01-15 11:57:03 +0100 | [diff] [blame] | 173 | unsigned long fd_cache_mask = 0; // Mask of threads with events in the cache |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 174 | |
| 175 | THREAD_LOCAL int *fd_updt = NULL; // FD updates list |
| 176 | THREAD_LOCAL int fd_nbupdt = 0; // number of updates in the list |
| 177 | |
Willy Tarreau | 4cc67a2 | 2018-02-05 17:14:55 +0100 | [diff] [blame^] | 178 | /* adds fd <fd> to fd list <list> if it was not yet in it */ |
| 179 | void fd_add_to_fd_list(volatile struct fdlist *list, int fd) |
| 180 | { |
| 181 | int next; |
| 182 | int new; |
| 183 | int old; |
| 184 | int last; |
| 185 | |
| 186 | redo_next: |
| 187 | next = fdtab[fd].cache.next; |
| 188 | /* Check that we're not already in the cache, and if not, lock us. */ |
| 189 | if (next >= -2) |
| 190 | goto done; |
| 191 | if (!HA_ATOMIC_CAS(&fdtab[fd].cache.next, &next, -2)) |
| 192 | goto redo_next; |
| 193 | __ha_barrier_store(); |
| 194 | redo_last: |
| 195 | /* First, insert in the linked list */ |
| 196 | last = list->last; |
| 197 | old = -1; |
| 198 | new = fd; |
| 199 | if (unlikely(last == -1)) { |
| 200 | /* list is empty, try to add ourselves alone so that list->last=fd */ |
| 201 | |
| 202 | fdtab[fd].cache.prev = last; |
| 203 | |
| 204 | /* Make sure the "prev" store is visible before we update the last entry */ |
| 205 | __ha_barrier_store(); |
| 206 | if (unlikely(!HA_ATOMIC_CAS(&list->last, &old, new))) |
| 207 | goto redo_last; |
| 208 | |
| 209 | /* list->first was necessary -1, we're guaranteed to be alone here */ |
| 210 | list->first = fd; |
| 211 | |
| 212 | /* since we're alone at the end of the list and still locked(-2), |
| 213 | * we know noone tried to add past us. Mark the end of list. |
| 214 | */ |
| 215 | fdtab[fd].cache.next = -1; |
| 216 | goto done; /* We're done ! */ |
| 217 | } else { |
| 218 | /* non-empty list, add past the tail */ |
| 219 | do { |
| 220 | new = fd; |
| 221 | old = -1; |
| 222 | fdtab[fd].cache.prev = last; |
| 223 | |
| 224 | __ha_barrier_store(); |
| 225 | |
| 226 | /* adding ourselves past the last element |
| 227 | * The CAS will only succeed if its next is -1, |
| 228 | * which means it's in the cache, and the last element. |
| 229 | */ |
| 230 | if (likely(HA_ATOMIC_CAS(&fdtab[last].cache.next, &old, new))) |
| 231 | break; |
| 232 | goto redo_last; |
| 233 | } while (1); |
| 234 | } |
| 235 | /* Then, update the last entry */ |
| 236 | redo_fd_cache: |
| 237 | last = list->last; |
| 238 | __ha_barrier_load(); |
| 239 | |
| 240 | if (unlikely(!HA_ATOMIC_CAS(&list->last, &last, fd))) |
| 241 | goto redo_fd_cache; |
| 242 | __ha_barrier_store(); |
| 243 | fdtab[fd].cache.next = -1; |
| 244 | __ha_barrier_store(); |
| 245 | done: |
| 246 | return; |
| 247 | } |
| 248 | |
| 249 | /* removes fd <fd> from fd list <list> */ |
| 250 | void fd_rm_from_fd_list(volatile struct fdlist *list, int fd) |
| 251 | { |
| 252 | #if defined(HA_HAVE_CAS_DW) || defined(HA_CAS_IS_8B) |
| 253 | volatile struct fdlist_entry cur_list, next_list; |
| 254 | #endif |
| 255 | int old; |
| 256 | int new = -2; |
| 257 | int prev; |
| 258 | int next; |
| 259 | int last; |
| 260 | |
| 261 | lock_self: |
| 262 | #if (defined(HA_CAS_IS_8B) || defined(HA_HAVE_CAS_DW)) |
| 263 | next_list.next = next_list.prev = -2; |
| 264 | cur_list = fdtab[fd].cache; |
| 265 | /* First, attempt to lock our own entries */ |
| 266 | do { |
| 267 | /* The FD is not in the FD cache, give up */ |
| 268 | if (unlikely(cur_list.next <= -3)) |
| 269 | return; |
| 270 | if (unlikely(cur_list.prev == -2 || cur_list.next == -2)) |
| 271 | goto lock_self; |
| 272 | } while ( |
| 273 | #ifdef HA_CAS_IS_8B |
| 274 | unlikely(!HA_ATOMIC_CAS(((void **)(void *)&fdtab[fd].cache.next), ((void **)(void *)&cur_list), (*(void **)(void *)&next_list)))) |
| 275 | #else |
| 276 | unlikely(!__ha_cas_dw((void *)&fdtab[fd].cache.next, (void *)&cur_list, (void *)&next_list))) |
| 277 | #endif |
| 278 | ; |
| 279 | next = cur_list.next; |
| 280 | prev = cur_list.prev; |
| 281 | |
| 282 | #else |
| 283 | lock_self_next: |
| 284 | next = fdtab[fd].cache.next; |
| 285 | if (next == -2) |
| 286 | goto lock_self_next; |
| 287 | if (next <= -3) |
| 288 | goto done; |
| 289 | if (unlikely(!HA_ATOMIC_CAS(&fdtab[fd].cache.next, &next, -2))) |
| 290 | goto lock_self_next; |
| 291 | lock_self_prev: |
| 292 | prev = fdtab[fd].cache.prev; |
| 293 | if (prev == -2) |
| 294 | goto lock_self_prev; |
| 295 | if (unlikely(!HA_ATOMIC_CAS(&fdtab[fd].cache.prev, &prev, -2))) |
| 296 | goto lock_self_prev; |
| 297 | #endif |
| 298 | __ha_barrier_store(); |
| 299 | |
| 300 | /* Now, lock the entries of our neighbours */ |
| 301 | if (likely(prev != -1)) { |
| 302 | redo_prev: |
| 303 | old = fd; |
| 304 | |
| 305 | if (unlikely(!HA_ATOMIC_CAS(&fdtab[prev].cache.next, &old, new))) { |
| 306 | if (unlikely(old == -2)) { |
| 307 | /* Neighbour already locked, give up and |
| 308 | * retry again once he's done |
| 309 | */ |
| 310 | fdtab[fd].cache.prev = prev; |
| 311 | __ha_barrier_store(); |
| 312 | fdtab[fd].cache.next = next; |
| 313 | __ha_barrier_store(); |
| 314 | goto lock_self; |
| 315 | } |
| 316 | goto redo_prev; |
| 317 | } |
| 318 | } |
| 319 | if (likely(next != -1)) { |
| 320 | redo_next: |
| 321 | old = fd; |
| 322 | if (unlikely(!HA_ATOMIC_CAS(&fdtab[next].cache.prev, &old, new))) { |
| 323 | if (unlikely(old == -2)) { |
| 324 | /* Neighbour already locked, give up and |
| 325 | * retry again once he's done |
| 326 | */ |
| 327 | if (prev != -1) { |
| 328 | fdtab[prev].cache.next = fd; |
| 329 | __ha_barrier_store(); |
| 330 | } |
| 331 | fdtab[fd].cache.prev = prev; |
| 332 | __ha_barrier_store(); |
| 333 | fdtab[fd].cache.next = next; |
| 334 | __ha_barrier_store(); |
| 335 | goto lock_self; |
| 336 | } |
| 337 | goto redo_next; |
| 338 | } |
| 339 | } |
| 340 | if (list->first == fd) |
| 341 | list->first = next; |
| 342 | __ha_barrier_store(); |
| 343 | last = list->last; |
| 344 | while (unlikely(last == fd && (!HA_ATOMIC_CAS(&list->last, &last, prev)))) |
| 345 | __ha_compiler_barrier(); |
| 346 | /* Make sure we let other threads know we're no longer in cache, |
| 347 | * before releasing our neighbours. |
| 348 | */ |
| 349 | __ha_barrier_store(); |
| 350 | if (likely(prev != -1)) |
| 351 | fdtab[prev].cache.next = next; |
| 352 | __ha_barrier_store(); |
| 353 | if (likely(next != -1)) |
| 354 | fdtab[next].cache.prev = prev; |
| 355 | __ha_barrier_store(); |
| 356 | /* Ok, now we're out of the fd cache */ |
| 357 | fdtab[fd].cache.next = -(next + 4); |
| 358 | __ha_barrier_store(); |
| 359 | done: |
| 360 | return; |
| 361 | } |
| 362 | |
Willy Tarreau | 173d995 | 2018-01-26 21:48:23 +0100 | [diff] [blame] | 363 | /* Deletes an FD from the fdsets. |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 364 | * The file descriptor is also closed. |
| 365 | */ |
Olivier Houchard | 1fc0516 | 2017-04-06 01:05:05 +0200 | [diff] [blame] | 366 | static void fd_dodelete(int fd, int do_close) |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 367 | { |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 368 | HA_SPIN_LOCK(FD_LOCK, &fdtab[fd].lock); |
Willy Tarreau | ad38ace | 2013-12-15 14:19:38 +0100 | [diff] [blame] | 369 | if (fdtab[fd].linger_risk) { |
| 370 | /* this is generally set when connecting to servers */ |
| 371 | setsockopt(fd, SOL_SOCKET, SO_LINGER, |
| 372 | (struct linger *) &nolinger, sizeof(struct linger)); |
| 373 | } |
Willy Tarreau | 6ea20b1 | 2012-11-11 16:05:19 +0100 | [diff] [blame] | 374 | if (cur_poller.clo) |
| 375 | cur_poller.clo(fd); |
| 376 | |
Willy Tarreau | 899d957 | 2014-01-25 19:20:35 +0100 | [diff] [blame] | 377 | fd_release_cache_entry(fd); |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 378 | fdtab[fd].state = 0; |
Willy Tarreau | 6ea20b1 | 2012-11-11 16:05:19 +0100 | [diff] [blame] | 379 | |
Willy Tarreau | 8d5d77e | 2009-10-18 07:25:52 +0200 | [diff] [blame] | 380 | port_range_release_port(fdinfo[fd].port_range, fdinfo[fd].local_port); |
| 381 | fdinfo[fd].port_range = NULL; |
Willy Tarreau | db3b326 | 2012-07-05 23:19:22 +0200 | [diff] [blame] | 382 | fdtab[fd].owner = NULL; |
Willy Tarreau | ebc78d7 | 2018-01-20 23:53:50 +0100 | [diff] [blame] | 383 | fdtab[fd].update_mask &= ~tid_bit; |
Willy Tarreau | f65610a | 2017-10-31 16:06:06 +0100 | [diff] [blame] | 384 | fdtab[fd].thread_mask = 0; |
Willy Tarreau | c9c8378 | 2018-01-17 18:44:46 +0100 | [diff] [blame] | 385 | if (do_close) { |
| 386 | fdtab[fd].polled_mask = 0; |
Christopher Faulet | d531f88 | 2017-06-01 16:55:03 +0200 | [diff] [blame] | 387 | close(fd); |
Willy Tarreau | c9c8378 | 2018-01-17 18:44:46 +0100 | [diff] [blame] | 388 | } |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 389 | HA_SPIN_UNLOCK(FD_LOCK, &fdtab[fd].lock); |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 390 | } |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 391 | |
Willy Tarreau | 173d995 | 2018-01-26 21:48:23 +0100 | [diff] [blame] | 392 | /* Deletes an FD from the fdsets. |
Olivier Houchard | 1fc0516 | 2017-04-06 01:05:05 +0200 | [diff] [blame] | 393 | * The file descriptor is also closed. |
| 394 | */ |
| 395 | void fd_delete(int fd) |
| 396 | { |
| 397 | fd_dodelete(fd, 1); |
| 398 | } |
| 399 | |
Willy Tarreau | 173d995 | 2018-01-26 21:48:23 +0100 | [diff] [blame] | 400 | /* Deletes an FD from the fdsets. |
Olivier Houchard | 1fc0516 | 2017-04-06 01:05:05 +0200 | [diff] [blame] | 401 | * The file descriptor is kept open. |
| 402 | */ |
| 403 | void fd_remove(int fd) |
| 404 | { |
| 405 | fd_dodelete(fd, 0); |
| 406 | } |
| 407 | |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 408 | static inline void fdlist_process_cached_events(volatile struct fdlist *fdlist) |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 409 | { |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 410 | int fd, old_fd, e; |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 411 | |
Olivier Houchard | 1256836 | 2018-01-31 18:07:29 +0100 | [diff] [blame] | 412 | for (old_fd = fd = fdlist->first; fd != -1; fd = fdtab[fd].cache.next) { |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 413 | if (fd == -2) { |
| 414 | fd = old_fd; |
| 415 | continue; |
| 416 | } else if (fd <= -3) |
| 417 | fd = -fd - 4; |
| 418 | if (fd == -1) |
| 419 | break; |
| 420 | old_fd = fd; |
| 421 | if (!(fdtab[fd].thread_mask & tid_bit)) |
| 422 | continue; |
Olivier Houchard | 1256836 | 2018-01-31 18:07:29 +0100 | [diff] [blame] | 423 | if (fdtab[fd].cache.next < -3) |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 424 | continue; |
Christopher Faulet | 69553fe | 2018-01-15 11:57:03 +0100 | [diff] [blame] | 425 | |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 426 | HA_ATOMIC_OR(&fd_cache_mask, tid_bit); |
Willy Tarreau | d80cb4e | 2018-01-20 19:30:13 +0100 | [diff] [blame] | 427 | if (HA_SPIN_TRYLOCK(FD_LOCK, &fdtab[fd].lock)) { |
| 428 | activity[tid].fd_lock++; |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 429 | continue; |
Willy Tarreau | d80cb4e | 2018-01-20 19:30:13 +0100 | [diff] [blame] | 430 | } |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 431 | |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 432 | e = fdtab[fd].state; |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 433 | fdtab[fd].ev &= FD_POLL_STICKY; |
| 434 | |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 435 | if ((e & (FD_EV_READY_R | FD_EV_ACTIVE_R)) == (FD_EV_READY_R | FD_EV_ACTIVE_R)) |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 436 | fdtab[fd].ev |= FD_POLL_IN; |
| 437 | |
Willy Tarreau | f817e9f | 2014-01-10 16:58:45 +0100 | [diff] [blame] | 438 | if ((e & (FD_EV_READY_W | FD_EV_ACTIVE_W)) == (FD_EV_READY_W | FD_EV_ACTIVE_W)) |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 439 | fdtab[fd].ev |= FD_POLL_OUT; |
| 440 | |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 441 | if (fdtab[fd].iocb && fdtab[fd].owner && fdtab[fd].ev) { |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 442 | HA_SPIN_UNLOCK(FD_LOCK, &fdtab[fd].lock); |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 443 | fdtab[fd].iocb(fd); |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 444 | } |
| 445 | else { |
Willy Tarreau | 5be2f35 | 2014-11-19 19:43:05 +0100 | [diff] [blame] | 446 | fd_release_cache_entry(fd); |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 447 | HA_SPIN_UNLOCK(FD_LOCK, &fdtab[fd].lock); |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 448 | } |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 449 | } |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 450 | } |
| 451 | |
| 452 | /* Scan and process the cached events. This should be called right after |
| 453 | * the poller. The loop may cause new entries to be created, for example |
| 454 | * if a listener causes an accept() to initiate a new incoming connection |
| 455 | * wanting to attempt an recv(). |
| 456 | */ |
| 457 | void fd_process_cached_events() |
| 458 | { |
| 459 | HA_ATOMIC_AND(&fd_cache_mask, ~tid_bit); |
| 460 | fdlist_process_cached_events(&fd_cache_local[tid]); |
| 461 | fdlist_process_cached_events(&fd_cache); |
Willy Tarreau | 09f2456 | 2012-11-11 16:43:45 +0100 | [diff] [blame] | 462 | } |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 463 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 464 | /* disable the specified poller */ |
| 465 | void disable_poller(const char *poller_name) |
| 466 | { |
| 467 | int p; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 468 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 469 | for (p = 0; p < nbpollers; p++) |
| 470 | if (strcmp(pollers[p].name, poller_name) == 0) |
| 471 | pollers[p].pref = 0; |
| 472 | } |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 473 | |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 474 | /* Initialize the pollers per thread */ |
| 475 | static int init_pollers_per_thread() |
| 476 | { |
| 477 | if ((fd_updt = calloc(global.maxsock, sizeof(*fd_updt))) == NULL) |
| 478 | return 0; |
| 479 | return 1; |
| 480 | } |
| 481 | |
| 482 | /* Deinitialize the pollers per thread */ |
| 483 | static void deinit_pollers_per_thread() |
| 484 | { |
| 485 | free(fd_updt); |
| 486 | fd_updt = NULL; |
| 487 | } |
| 488 | |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 489 | /* |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 490 | * Initialize the pollers till the best one is found. |
| 491 | * If none works, returns 0, otherwise 1. |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 492 | */ |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 493 | int init_pollers() |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 494 | { |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 495 | int p; |
| 496 | struct poller *bp; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 497 | |
Christopher Faulet | 63fe652 | 2017-08-31 17:52:09 +0200 | [diff] [blame] | 498 | if ((fdtab = calloc(global.maxsock, sizeof(struct fdtab))) == NULL) |
| 499 | goto fail_tab; |
| 500 | |
| 501 | if ((fdinfo = calloc(global.maxsock, sizeof(struct fdinfo))) == NULL) |
| 502 | goto fail_info; |
| 503 | |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 504 | fd_cache.first = fd_cache.last = -1; |
Christopher Faulet | cd7879a | 2017-10-27 13:53:47 +0200 | [diff] [blame] | 505 | hap_register_per_thread_init(init_pollers_per_thread); |
| 506 | hap_register_per_thread_deinit(deinit_pollers_per_thread); |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 507 | |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 508 | for (p = 0; p < global.maxsock; p++) { |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 509 | HA_SPIN_INIT(&fdtab[p].lock); |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 510 | /* Mark the fd as out of the fd cache */ |
Olivier Houchard | 1256836 | 2018-01-31 18:07:29 +0100 | [diff] [blame] | 511 | fdtab[p].cache.next = -3; |
| 512 | fdtab[p].cache.next = -3; |
Olivier Houchard | 4815c8c | 2018-01-24 18:17:56 +0100 | [diff] [blame] | 513 | } |
| 514 | for (p = 0; p < global.nbthread; p++) |
| 515 | fd_cache_local[p].first = fd_cache_local[p].last = -1; |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 516 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 517 | do { |
| 518 | bp = NULL; |
| 519 | for (p = 0; p < nbpollers; p++) |
| 520 | if (!bp || (pollers[p].pref > bp->pref)) |
| 521 | bp = &pollers[p]; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 522 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 523 | if (!bp || bp->pref == 0) |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 524 | break; |
| 525 | |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 526 | if (bp->init(bp)) { |
| 527 | memcpy(&cur_poller, bp, sizeof(*bp)); |
| 528 | return 1; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 529 | } |
Willy Tarreau | 4f60f16 | 2007-04-08 16:39:58 +0200 | [diff] [blame] | 530 | } while (!bp || bp->pref == 0); |
| 531 | return 0; |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 532 | |
Willy Tarreau | 16f649c | 2014-01-25 19:10:48 +0100 | [diff] [blame] | 533 | fail_cache: |
Christopher Faulet | 63fe652 | 2017-08-31 17:52:09 +0200 | [diff] [blame] | 534 | free(fdinfo); |
| 535 | fail_info: |
| 536 | free(fdtab); |
| 537 | fail_tab: |
Willy Tarreau | 7be79a4 | 2012-11-11 15:02:54 +0100 | [diff] [blame] | 538 | return 0; |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 539 | } |
| 540 | |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 541 | /* |
Krzysztof Piotr Oledzki | a643baf | 2008-05-29 23:53:44 +0200 | [diff] [blame] | 542 | * Deinitialize the pollers. |
| 543 | */ |
| 544 | void deinit_pollers() { |
| 545 | |
| 546 | struct poller *bp; |
| 547 | int p; |
| 548 | |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 549 | for (p = 0; p < global.maxsock; p++) |
Christopher Faulet | 2a944ee | 2017-11-07 10:42:54 +0100 | [diff] [blame] | 550 | HA_SPIN_DESTROY(&fdtab[p].lock); |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 551 | |
Krzysztof Piotr Oledzki | a643baf | 2008-05-29 23:53:44 +0200 | [diff] [blame] | 552 | for (p = 0; p < nbpollers; p++) { |
| 553 | bp = &pollers[p]; |
| 554 | |
| 555 | if (bp && bp->pref) |
| 556 | bp->term(bp); |
| 557 | } |
Christopher Faulet | d4604ad | 2017-05-29 10:40:41 +0200 | [diff] [blame] | 558 | |
Christopher Faulet | 63fe652 | 2017-08-31 17:52:09 +0200 | [diff] [blame] | 559 | free(fdinfo); fdinfo = NULL; |
| 560 | free(fdtab); fdtab = NULL; |
Krzysztof Piotr Oledzki | a643baf | 2008-05-29 23:53:44 +0200 | [diff] [blame] | 561 | } |
| 562 | |
| 563 | /* |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 564 | * Lists the known pollers on <out>. |
| 565 | * Should be performed only before initialization. |
| 566 | */ |
| 567 | int list_pollers(FILE *out) |
| 568 | { |
| 569 | int p; |
| 570 | int last, next; |
| 571 | int usable; |
| 572 | struct poller *bp; |
| 573 | |
| 574 | fprintf(out, "Available polling systems :\n"); |
| 575 | |
| 576 | usable = 0; |
| 577 | bp = NULL; |
| 578 | last = next = -1; |
| 579 | while (1) { |
| 580 | for (p = 0; p < nbpollers; p++) { |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 581 | if ((next < 0 || pollers[p].pref > next) |
Willy Tarreau | e79c3b2 | 2010-11-19 10:20:36 +0100 | [diff] [blame] | 582 | && (last < 0 || pollers[p].pref < last)) { |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 583 | next = pollers[p].pref; |
Willy Tarreau | e79c3b2 | 2010-11-19 10:20:36 +0100 | [diff] [blame] | 584 | if (!bp || (pollers[p].pref > bp->pref)) |
| 585 | bp = &pollers[p]; |
| 586 | } |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 587 | } |
| 588 | |
| 589 | if (next == -1) |
| 590 | break; |
| 591 | |
| 592 | for (p = 0; p < nbpollers; p++) { |
| 593 | if (pollers[p].pref == next) { |
| 594 | fprintf(out, " %10s : ", pollers[p].name); |
| 595 | if (pollers[p].pref == 0) |
| 596 | fprintf(out, "disabled, "); |
| 597 | else |
| 598 | fprintf(out, "pref=%3d, ", pollers[p].pref); |
| 599 | if (pollers[p].test(&pollers[p])) { |
| 600 | fprintf(out, " test result OK"); |
| 601 | if (next > 0) |
| 602 | usable++; |
Willy Tarreau | e79c3b2 | 2010-11-19 10:20:36 +0100 | [diff] [blame] | 603 | } else { |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 604 | fprintf(out, " test result FAILED"); |
Willy Tarreau | e79c3b2 | 2010-11-19 10:20:36 +0100 | [diff] [blame] | 605 | if (bp == &pollers[p]) |
| 606 | bp = NULL; |
| 607 | } |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 608 | fprintf(out, "\n"); |
| 609 | } |
| 610 | } |
| 611 | last = next; |
| 612 | next = -1; |
| 613 | }; |
| 614 | fprintf(out, "Total: %d (%d usable), will use %s.\n", nbpollers, usable, bp ? bp->name : "none"); |
| 615 | return 0; |
| 616 | } |
| 617 | |
| 618 | /* |
| 619 | * Some pollers may lose their connection after a fork(). It may be necessary |
| 620 | * to create initialize part of them again. Returns 0 in case of failure, |
| 621 | * otherwise 1. The fork() function may be NULL if unused. In case of error, |
| 622 | * the the current poller is destroyed and the caller is responsible for trying |
| 623 | * another one by calling init_pollers() again. |
| 624 | */ |
| 625 | int fork_poller() |
| 626 | { |
Conrad Hoffmann | 041751c | 2014-05-20 14:28:24 +0200 | [diff] [blame] | 627 | int fd; |
Willy Tarreau | ce036bc | 2018-01-29 14:58:02 +0100 | [diff] [blame] | 628 | for (fd = 0; fd < global.maxsock; fd++) { |
Conrad Hoffmann | 041751c | 2014-05-20 14:28:24 +0200 | [diff] [blame] | 629 | if (fdtab[fd].owner) { |
| 630 | fdtab[fd].cloned = 1; |
| 631 | } |
| 632 | } |
| 633 | |
Willy Tarreau | 2ff7622 | 2007-04-09 19:29:56 +0200 | [diff] [blame] | 634 | if (cur_poller.fork) { |
| 635 | if (cur_poller.fork(&cur_poller)) |
| 636 | return 1; |
| 637 | cur_poller.term(&cur_poller); |
| 638 | return 0; |
| 639 | } |
| 640 | return 1; |
| 641 | } |
| 642 | |
| 643 | /* |
Willy Tarreau | baaee00 | 2006-06-26 02:48:02 +0200 | [diff] [blame] | 644 | * Local variables: |
| 645 | * c-indent-level: 8 |
| 646 | * c-basic-offset: 8 |
| 647 | * End: |
| 648 | */ |