Paul Beesley | 236d246 | 2019-03-05 17:19:37 +0000 | [diff] [blame] | 1 | PSCI Performance Measurements on Arm Juno Development Platform |
| 2 | ============================================================== |
| 3 | |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 4 | This document summarises the findings of performance measurements of key |
John Tsichritzis | 63801cd | 2019-07-05 14:22:12 +0100 | [diff] [blame] | 5 | operations in the Trusted Firmware-A Power State Coordination Interface (PSCI) |
| 6 | implementation, using the in-built Performance Measurement Framework (PMF) and |
| 7 | runtime instrumentation timestamps. |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 8 | |
| 9 | Method |
| 10 | ------ |
| 11 | |
| 12 | We used the `Juno R1 platform`_ for these tests, which has 4 x Cortex-A53 and 2 |
| 13 | x Cortex-A57 clusters running at the following frequencies: |
| 14 | |
| 15 | +-----------------+--------------------+ |
| 16 | | Domain | Frequency (MHz) | |
| 17 | +=================+====================+ |
| 18 | | Cortex-A57 | 900 (nominal) | |
| 19 | +-----------------+--------------------+ |
| 20 | | Cortex-A53 | 650 (underdrive) | |
| 21 | +-----------------+--------------------+ |
| 22 | | AXI subsystem | 533 | |
| 23 | +-----------------+--------------------+ |
| 24 | |
| 25 | Juno supports CPU, cluster and system power down states, corresponding to power |
| 26 | levels 0, 1 and 2 respectively. It does not support any retention states. |
| 27 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 28 | Given that runtime instrumentation using PMF is invasive, there is a small |
| 29 | (unquantified) overhead on the results. PMF uses the generic counter for |
| 30 | timestamps, which runs at 50MHz on Juno. |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 31 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 32 | The following source trees and binaries were used: |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 33 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 34 | - TF-A [`v2.9-rc0`_] |
| 35 | - TFTF [`v2.9-rc0`_] |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 36 | |
Thaddeus Serna | 8709cc9 | 2023-08-14 13:28:59 -0500 | [diff] [blame] | 37 | Please see the Runtime Instrumentation :ref:`Testing Methodology |
| 38 | <Runtime Instrumentation Methodology>` |
| 39 | page for more details. |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 40 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 41 | Procedure |
| 42 | --------- |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 43 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 44 | #. Build TFTF with runtime instrumentation enabled: |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 45 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 46 | .. code:: shell |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 47 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 48 | make CROSS_COMPILE=aarch64-none-elf- PLAT=juno \ |
| 49 | TESTS=runtime-instrumentation all |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 50 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 51 | #. Fetch Juno's SCP binary from TF-A's archive: |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 52 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 53 | .. code:: shell |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 54 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 55 | curl --fail --connect-timeout 5 --retry 5 -sLS -o scp_bl2.bin \ |
| 56 | https://downloads.trustedfirmware.org/tf-a/css_scp_2.12.0/juno/release/juno-bl2.bin |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 57 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 58 | #. Build TF-A with the following build options: |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 59 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 60 | .. code:: shell |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 61 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 62 | make CROSS_COMPILE=aarch64-none-elf- PLAT=juno \ |
| 63 | BL33="/path/to/tftf.bin" SCP_BL2="scp_bl2.bin" \ |
| 64 | ENABLE_RUNTIME_INSTRUMENTATION=1 fiptool all fip |
| 65 | |
| 66 | #. Load the following images onto the development board: ``fip.bin``, |
| 67 | ``scp_bl2.bin``. |
| 68 | |
| 69 | Results |
| 70 | ------- |
| 71 | |
| 72 | ``CPU_SUSPEND`` to deepest power level |
| 73 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 74 | |
| 75 | .. table:: ``CPU_SUSPEND`` latencies (µs) to deepest power level in |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 76 | parallel (v2.9) |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 77 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 78 | +---------+------+-----------+--------+-------------+ |
| 79 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 80 | +---------+------+-----------+--------+-------------+ |
| 81 | | 0 | 0 | 104.58 | 241.20 | 5.26 | |
| 82 | +---------+------+-----------+--------+-------------+ |
| 83 | | 0 | 1 | 384.24 | 22.50 | 138.76 | |
| 84 | +---------+------+-----------+--------+-------------+ |
| 85 | | 1 | 0 | 244.56 | 22.18 | 5.16 | |
| 86 | +---------+------+-----------+--------+-------------+ |
| 87 | | 1 | 1 | 670.56 | 18.58 | 4.44 | |
| 88 | +---------+------+-----------+--------+-------------+ |
| 89 | | 1 | 2 | 809.36 | 269.28 | 4.44 | |
| 90 | +---------+------+-----------+--------+-------------+ |
| 91 | | 1 | 3 | 984.96 | 219.70 | 79.62 | |
| 92 | +---------+------+-----------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 93 | |
| 94 | .. table:: ``CPU_SUSPEND`` latencies (µs) to deepest power level in |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 95 | parallel (v2.10) |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 96 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 97 | +---------+------+-------------------+--------+-------------+ |
| 98 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 99 | +---------+------+-------------------+--------+-------------+ |
| 100 | | 0 | 0 | 242.66 (+132.03%) | 245.1 | 5.4 | |
| 101 | +---------+------+-------------------+--------+-------------+ |
| 102 | | 0 | 1 | 522.08 (+35.87%) | 26.24 | 138.32 | |
| 103 | +---------+------+-------------------+--------+-------------+ |
| 104 | | 1 | 0 | 104.36 (-57.33%) | 27.1 | 5.32 | |
| 105 | +---------+------+-------------------+--------+-------------+ |
| 106 | | 1 | 1 | 382.56 (-42.95%) | 23.34 | 4.42 | |
| 107 | +---------+------+-------------------+--------+-------------+ |
| 108 | | 1 | 2 | 807.74 | 271.54 | 4.64 | |
| 109 | +---------+------+-------------------+--------+-------------+ |
| 110 | | 1 | 3 | 981.36 | 221.8 | 79.48 | |
| 111 | +---------+------+-------------------+--------+-------------+ |
| 112 | |
| 113 | .. table:: ``CPU_SUSPEND`` latencies (µs) to deepest power level in |
| 114 | serial (v2.9) |
| 115 | |
| 116 | +---------+------+-----------+--------+-------------+ |
| 117 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 118 | +---------+------+-----------+--------+-------------+ |
| 119 | | 0 | 0 | 236.56 | 23.24 | 138.18 | |
| 120 | +---------+------+-----------+--------+-------------+ |
| 121 | | 0 | 1 | 236.86 | 23.28 | 138.10 | |
| 122 | +---------+------+-----------+--------+-------------+ |
| 123 | | 1 | 0 | 281.04 | 22.80 | 77.24 | |
| 124 | +---------+------+-----------+--------+-------------+ |
| 125 | | 1 | 1 | 100.28 | 18.52 | 4.54 | |
| 126 | +---------+------+-----------+--------+-------------+ |
| 127 | | 1 | 2 | 100.12 | 18.78 | 4.50 | |
| 128 | +---------+------+-----------+--------+-------------+ |
| 129 | | 1 | 3 | 100.36 | 18.94 | 4.44 | |
| 130 | +---------+------+-----------+--------+-------------+ |
| 131 | |
| 132 | .. table:: ``CPU_SUSPEND`` latencies (µs) to deepest power level in |
| 133 | serial (v2.10) |
| 134 | |
| 135 | +---------+------+-----------+--------+-------------+ |
| 136 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 137 | +---------+------+-----------+--------+-------------+ |
| 138 | | 0 | 0 | 236.84 | 27.1 | 138.36 | |
| 139 | +---------+------+-----------+--------+-------------+ |
| 140 | | 0 | 1 | 236.96 | 27.1 | 138.32 | |
| 141 | +---------+------+-----------+--------+-------------+ |
| 142 | | 1 | 0 | 280.06 | 26.94 | 77.5 | |
| 143 | +---------+------+-----------+--------+-------------+ |
| 144 | | 1 | 1 | 100.76 | 23.42 | 4.36 | |
| 145 | +---------+------+-----------+--------+-------------+ |
| 146 | | 1 | 2 | 100.02 | 23.42 | 4.44 | |
| 147 | +---------+------+-----------+--------+-------------+ |
| 148 | | 1 | 3 | 100.08 | 23.2 | 4.4 | |
| 149 | +---------+------+-----------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 150 | |
| 151 | ``CPU_SUSPEND`` to power level 0 |
| 152 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 153 | |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 154 | .. table:: ``CPU_SUSPEND`` latencies (µs) to power level 0 in |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 155 | parallel (v2.9) |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 156 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 157 | +---------+------+-----------+--------+-------------+ |
| 158 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 159 | +---------+------+-----------+--------+-------------+ |
| 160 | | 0 | 0 | 662.34 | 15.22 | 8.08 | |
| 161 | +---------+------+-----------+--------+-------------+ |
| 162 | | 0 | 1 | 802.00 | 15.50 | 8.16 | |
| 163 | +---------+------+-----------+--------+-------------+ |
| 164 | | 1 | 0 | 385.22 | 15.74 | 7.88 | |
| 165 | +---------+------+-----------+--------+-------------+ |
| 166 | | 1 | 1 | 106.16 | 16.06 | 7.44 | |
| 167 | +---------+------+-----------+--------+-------------+ |
| 168 | | 1 | 2 | 524.38 | 15.64 | 7.34 | |
| 169 | +---------+------+-----------+--------+-------------+ |
| 170 | | 1 | 3 | 246.00 | 15.78 | 7.72 | |
| 171 | +---------+------+-----------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 172 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 173 | .. table:: ``CPU_SUSPEND`` latencies (µs) to power level 0 in |
| 174 | parallel (v2.10) |
| 175 | |
| 176 | +---------+------+-------------------+--------+-------------+ |
| 177 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 178 | +---------+------+-------------------+--------+-------------+ |
| 179 | | 0 | 0 | 801.04 | 18.66 | 8.22 | |
| 180 | +---------+------+-------------------+--------+-------------+ |
| 181 | | 0 | 1 | 661.28 | 19.08 | 7.88 | |
| 182 | +---------+------+-------------------+--------+-------------+ |
| 183 | | 1 | 0 | 105.9 (-72.51%) | 20.3 | 7.58 | |
| 184 | +---------+------+-------------------+--------+-------------+ |
| 185 | | 1 | 1 | 383.58 (+261.32%) | 20.4 | 7.42 | |
| 186 | +---------+------+-------------------+--------+-------------+ |
| 187 | | 1 | 2 | 523.52 | 20.1 | 7.74 | |
| 188 | +---------+------+-------------------+--------+-------------+ |
| 189 | | 1 | 3 | 244.5 | 20.16 | 7.56 | |
| 190 | +---------+------+-------------------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 191 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 192 | .. table:: ``CPU_SUSPEND`` latencies (µs) to power level 0 in serial (v2.9) |
| 193 | |
| 194 | +---------+------+-----------+--------+-------------+ |
| 195 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 196 | +---------+------+-----------+--------+-------------+ |
| 197 | | 0 | 0 | 99.80 | 15.94 | 5.42 | |
| 198 | +---------+------+-----------+--------+-------------+ |
| 199 | | 0 | 1 | 99.76 | 15.80 | 5.24 | |
| 200 | +---------+------+-----------+--------+-------------+ |
| 201 | | 1 | 0 | 278.26 | 16.16 | 4.58 | |
| 202 | +---------+------+-----------+--------+-------------+ |
| 203 | | 1 | 1 | 96.88 | 16.00 | 4.52 | |
| 204 | +---------+------+-----------+--------+-------------+ |
| 205 | | 1 | 2 | 96.80 | 16.12 | 4.54 | |
| 206 | +---------+------+-----------+--------+-------------+ |
| 207 | | 1 | 3 | 96.88 | 16.12 | 4.54 | |
| 208 | +---------+------+-----------+--------+-------------+ |
| 209 | |
| 210 | .. table:: ``CPU_SUSPEND`` latencies (µs) to power level 0 in serial (v2.10) |
| 211 | |
| 212 | +---------+------+-----------+--------+-------------+ |
| 213 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 214 | +---------+------+-----------+--------+-------------+ |
| 215 | | 0 | 0 | 99.84 | 18.86 | 5.54 | |
| 216 | +---------+------+-----------+--------+-------------+ |
| 217 | | 0 | 1 | 100.2 | 18.82 | 5.66 | |
| 218 | +---------+------+-----------+--------+-------------+ |
| 219 | | 1 | 0 | 278.12 | 20.56 | 4.48 | |
| 220 | +---------+------+-----------+--------+-------------+ |
| 221 | | 1 | 1 | 96.68 | 20.62 | 4.3 | |
| 222 | +---------+------+-----------+--------+-------------+ |
| 223 | | 1 | 2 | 96.94 | 20.14 | 4.42 | |
| 224 | +---------+------+-----------+--------+-------------+ |
| 225 | | 1 | 3 | 96.68 | 20.46 | 4.32 | |
| 226 | +---------+------+-----------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 227 | |
| 228 | ``CPU_OFF`` on all non-lead CPUs |
| 229 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 230 | |
| 231 | ``CPU_OFF`` on all non-lead CPUs in sequence then, ``CPU_SUSPEND`` on the lead |
| 232 | core to the deepest power level. |
| 233 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 234 | .. table:: ``CPU_OFF`` latencies (µs) on all non-lead CPUs (v2.9) |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 235 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 236 | +---------+------+-----------+--------+-------------+ |
| 237 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 238 | +---------+------+-----------+--------+-------------+ |
| 239 | | 0 | 0 | 235.76 | 26.14 | 137.80 | |
| 240 | +---------+------+-----------+--------+-------------+ |
| 241 | | 0 | 1 | 235.40 | 25.72 | 137.62 | |
| 242 | +---------+------+-----------+--------+-------------+ |
| 243 | | 1 | 0 | 174.70 | 22.40 | 77.26 | |
| 244 | +---------+------+-----------+--------+-------------+ |
| 245 | | 1 | 1 | 100.92 | 24.04 | 4.52 | |
| 246 | +---------+------+-----------+--------+-------------+ |
| 247 | | 1 | 2 | 100.68 | 22.44 | 4.36 | |
| 248 | +---------+------+-----------+--------+-------------+ |
| 249 | | 1 | 3 | 101.36 | 22.70 | 4.52 | |
| 250 | +---------+------+-----------+--------+-------------+ |
| 251 | |
| 252 | .. table:: ``CPU_OFF`` latencies (µs) on all non-lead CPUs (v2.10) |
| 253 | |
| 254 | +---------------------------------------------------+ |
| 255 | | test_rt_instr_cpu_off_serial (latest) | |
| 256 | +---------+------+-----------+--------+-------------+ |
| 257 | | Cluster | Core | Powerdown | Wakeup | Cache Flush | |
| 258 | +---------+------+-----------+--------+-------------+ |
| 259 | | 0 | 0 | 236.04 | 30.02 | 137.9 | |
| 260 | +---------+------+-----------+--------+-------------+ |
| 261 | | 0 | 1 | 235.38 | 29.7 | 137.72 | |
| 262 | +---------+------+-----------+--------+-------------+ |
| 263 | | 1 | 0 | 175.18 | 26.96 | 77.26 | |
| 264 | +---------+------+-----------+--------+-------------+ |
| 265 | | 1 | 1 | 100.56 | 28.34 | 4.32 | |
| 266 | +---------+------+-----------+--------+-------------+ |
| 267 | | 1 | 2 | 100.38 | 26.82 | 4.3 | |
| 268 | +---------+------+-----------+--------+-------------+ |
| 269 | | 1 | 3 | 100.86 | 26.98 | 4.42 | |
| 270 | +---------+------+-----------+--------+-------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 271 | |
| 272 | ``CPU_VERSION`` in parallel |
| 273 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 274 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 275 | .. table:: ``CPU_VERSION`` latency (µs) in parallel on all cores (2.9) |
| 276 | |
| 277 | +-------------+--------+-------------+ |
| 278 | | Cluster | Core | Latency | |
| 279 | +-------------+--------+-------------+ |
| 280 | | 0 | 0 | 1.48 | |
| 281 | +-------------+--------+-------------+ |
| 282 | | 0 | 1 | 1.04 | |
| 283 | +-------------+--------+-------------+ |
| 284 | | 1 | 0 | 0.56 | |
| 285 | +-------------+--------+-------------+ |
| 286 | | 1 | 1 | 0.92 | |
| 287 | +-------------+--------+-------------+ |
| 288 | | 1 | 2 | 0.96 | |
| 289 | +-------------+--------+-------------+ |
| 290 | | 1 | 3 | 0.96 | |
| 291 | +-------------+--------+-------------+ |
| 292 | |
| 293 | .. table:: ``CPU_VERSION`` latency (µs) in parallel on all cores (2.10) |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 294 | |
Harrison Mutai | 9068845 | 2023-11-10 17:35:33 +0000 | [diff] [blame] | 295 | +-------------+--------+----------------------+ |
| 296 | | Cluster | Core | Latency | |
| 297 | +-------------+--------+----------------------+ |
| 298 | | 0 | 0 | 1.1 (-25.68%) | |
| 299 | +-------------+--------+----------------------+ |
| 300 | | 0 | 1 | 1.06 | |
| 301 | +-------------+--------+----------------------+ |
| 302 | | 1 | 0 | 0.58 | |
| 303 | +-------------+--------+----------------------+ |
| 304 | | 1 | 1 | 0.88 | |
| 305 | +-------------+--------+----------------------+ |
| 306 | | 1 | 2 | 0.92 | |
| 307 | +-------------+--------+----------------------+ |
| 308 | | 1 | 3 | 0.9 | |
| 309 | +-------------+--------+----------------------+ |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 310 | |
| 311 | Annotated Historic Results |
| 312 | -------------------------- |
| 313 | |
| 314 | The following results are based on the upstream `TF master as of 31/01/2017`_. |
| 315 | TF-A was built using the same build instructions as detailed in the procedure |
| 316 | above. |
| 317 | |
| 318 | In the results below, CPUs 0-3 refer to CPUs in the little cluster (A53) and |
| 319 | CPUs 4-5 refer to CPUs in the big cluster (A57). In all cases CPU 4 is the lead |
| 320 | CPU. |
| 321 | |
| 322 | ``PSCI_ENTRY`` corresponds to the powerdown latency, ``PSCI_EXIT`` the wakeup latency, and |
| 323 | ``CFLUSH_OVERHEAD`` the latency of the cache flush operation. |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 324 | |
| 325 | ``CPU_SUSPEND`` to deepest power level on all CPUs in parallel |
| 326 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 327 | |
| 328 | +-------+---------------------+--------------------+--------------------------+ |
| 329 | | CPU | ``PSCI_ENTRY`` (us) | ``PSCI_EXIT`` (us) | ``CFLUSH_OVERHEAD`` (us) | |
| 330 | +=======+=====================+====================+==========================+ |
| 331 | | 0 | 27 | 20 | 5 | |
| 332 | +-------+---------------------+--------------------+--------------------------+ |
| 333 | | 1 | 114 | 86 | 5 | |
| 334 | +-------+---------------------+--------------------+--------------------------+ |
| 335 | | 2 | 202 | 58 | 5 | |
| 336 | +-------+---------------------+--------------------+--------------------------+ |
| 337 | | 3 | 375 | 29 | 94 | |
| 338 | +-------+---------------------+--------------------+--------------------------+ |
| 339 | | 4 | 20 | 22 | 6 | |
| 340 | +-------+---------------------+--------------------+--------------------------+ |
| 341 | | 5 | 290 | 18 | 206 | |
| 342 | +-------+---------------------+--------------------+--------------------------+ |
| 343 | |
| 344 | A large variance in ``PSCI_ENTRY`` and ``PSCI_EXIT`` times across CPUs is |
| 345 | observed due to TF PSCI lock contention. In the worst case, CPU 3 has to wait |
| 346 | for the 3 other CPUs in the cluster (0-2) to complete ``PSCI_ENTRY`` and release |
| 347 | the lock before proceeding. |
| 348 | |
| 349 | The ``CFLUSH_OVERHEAD`` times for CPUs 3 and 5 are higher because they are the |
| 350 | last CPUs in their respective clusters to power down, therefore both the L1 and |
| 351 | L2 caches are flushed. |
| 352 | |
| 353 | The ``CFLUSH_OVERHEAD`` time for CPU 5 is a lot larger than that for CPU 3 |
| 354 | because the L2 cache size for the big cluster is lot larger (2MB) compared to |
| 355 | the little cluster (1MB). |
| 356 | |
| 357 | ``CPU_SUSPEND`` to power level 0 on all CPUs in parallel |
| 358 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 359 | |
| 360 | +-------+---------------------+--------------------+--------------------------+ |
| 361 | | CPU | ``PSCI_ENTRY`` (us) | ``PSCI_EXIT`` (us) | ``CFLUSH_OVERHEAD`` (us) | |
| 362 | +=======+=====================+====================+==========================+ |
| 363 | | 0 | 116 | 14 | 8 | |
| 364 | +-------+---------------------+--------------------+--------------------------+ |
| 365 | | 1 | 204 | 14 | 8 | |
| 366 | +-------+---------------------+--------------------+--------------------------+ |
| 367 | | 2 | 287 | 13 | 8 | |
| 368 | +-------+---------------------+--------------------+--------------------------+ |
| 369 | | 3 | 376 | 13 | 9 | |
| 370 | +-------+---------------------+--------------------+--------------------------+ |
| 371 | | 4 | 29 | 15 | 7 | |
| 372 | +-------+---------------------+--------------------+--------------------------+ |
| 373 | | 5 | 21 | 15 | 8 | |
| 374 | +-------+---------------------+--------------------+--------------------------+ |
| 375 | |
| 376 | There is no lock contention in TF generic code at power level 0 but the large |
| 377 | variance in ``PSCI_ENTRY`` times across CPUs is due to lock contention in Juno |
| 378 | platform code. The platform lock is used to mediate access to a single SCP |
| 379 | communication channel. This is compounded by the SCP firmware waiting for each |
| 380 | AP CPU to enter WFI before making the channel available to other CPUs, which |
| 381 | effectively serializes the SCP power down commands from all CPUs. |
| 382 | |
| 383 | On platforms with a more efficient CPU power down mechanism, it should be |
| 384 | possible to make the ``PSCI_ENTRY`` times smaller and consistent. |
| 385 | |
| 386 | The ``PSCI_EXIT`` times are consistent across all CPUs because TF does not |
| 387 | require locks at power level 0. |
| 388 | |
| 389 | The ``CFLUSH_OVERHEAD`` times for all CPUs are small and consistent since only |
| 390 | the cache associated with power level 0 is flushed (L1). |
| 391 | |
| 392 | ``CPU_SUSPEND`` to deepest power level on all CPUs in sequence |
| 393 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 394 | |
| 395 | +-------+---------------------+--------------------+--------------------------+ |
| 396 | | CPU | ``PSCI_ENTRY`` (us) | ``PSCI_EXIT`` (us) | ``CFLUSH_OVERHEAD`` (us) | |
| 397 | +=======+=====================+====================+==========================+ |
| 398 | | 0 | 114 | 20 | 94 | |
| 399 | +-------+---------------------+--------------------+--------------------------+ |
| 400 | | 1 | 114 | 20 | 94 | |
| 401 | +-------+---------------------+--------------------+--------------------------+ |
| 402 | | 2 | 114 | 20 | 94 | |
| 403 | +-------+---------------------+--------------------+--------------------------+ |
| 404 | | 3 | 114 | 20 | 94 | |
| 405 | +-------+---------------------+--------------------+--------------------------+ |
| 406 | | 4 | 195 | 22 | 180 | |
| 407 | +-------+---------------------+--------------------+--------------------------+ |
| 408 | | 5 | 21 | 17 | 6 | |
| 409 | +-------+---------------------+--------------------+--------------------------+ |
| 410 | |
Paul Beesley | f2ec714 | 2019-10-04 16:17:46 +0000 | [diff] [blame] | 411 | The ``CFLUSH_OVERHEAD`` times for lead CPU 4 and all CPUs in the non-lead cluster |
Joel Hutton | 9e60563 | 2019-02-25 15:18:56 +0000 | [diff] [blame] | 412 | are large because all other CPUs in the cluster are powered down during the |
| 413 | test. The ``CPU_SUSPEND`` call powers down to the cluster level, requiring a |
| 414 | flush of both L1 and L2 caches. |
| 415 | |
| 416 | The ``CFLUSH_OVERHEAD`` time for CPU 4 is a lot larger than those for the little |
| 417 | CPUs because the L2 cache size for the big cluster is lot larger (2MB) compared |
| 418 | to the little cluster (1MB). |
| 419 | |
| 420 | The ``PSCI_ENTRY`` and ``CFLUSH_OVERHEAD`` times for CPU 5 are low because lead |
| 421 | CPU 4 continues to run while CPU 5 is suspended. Hence CPU 5 only powers down to |
| 422 | level 0, which only requires L1 cache flush. |
| 423 | |
| 424 | ``CPU_SUSPEND`` to power level 0 on all CPUs in sequence |
| 425 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 426 | |
| 427 | +-------+---------------------+--------------------+--------------------------+ |
| 428 | | CPU | ``PSCI_ENTRY`` (us) | ``PSCI_EXIT`` (us) | ``CFLUSH_OVERHEAD`` (us) | |
| 429 | +=======+=====================+====================+==========================+ |
| 430 | | 0 | 22 | 14 | 5 | |
| 431 | +-------+---------------------+--------------------+--------------------------+ |
| 432 | | 1 | 22 | 14 | 5 | |
| 433 | +-------+---------------------+--------------------+--------------------------+ |
| 434 | | 2 | 21 | 14 | 5 | |
| 435 | +-------+---------------------+--------------------+--------------------------+ |
| 436 | | 3 | 22 | 14 | 5 | |
| 437 | +-------+---------------------+--------------------+--------------------------+ |
| 438 | | 4 | 17 | 14 | 6 | |
| 439 | +-------+---------------------+--------------------+--------------------------+ |
| 440 | | 5 | 18 | 15 | 6 | |
| 441 | +-------+---------------------+--------------------+--------------------------+ |
| 442 | |
| 443 | Here the times are small and consistent since there is no contention and it is |
| 444 | only necessary to flush the cache to power level 0 (L1). This is the best case |
| 445 | scenario. |
| 446 | |
| 447 | The ``PSCI_ENTRY`` times for CPUs in the big cluster are slightly smaller than |
| 448 | for the CPUs in little cluster due to greater CPU performance. |
| 449 | |
| 450 | The ``PSCI_EXIT`` times are generally lower than in the last test because the |
| 451 | cluster remains powered on throughout the test and there is less code to execute |
| 452 | on power on (for example, no need to enter CCI coherency) |
| 453 | |
| 454 | ``CPU_OFF`` on all non-lead CPUs in sequence then ``CPU_SUSPEND`` on lead CPU to deepest power level |
| 455 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 456 | |
| 457 | The test sequence here is as follows: |
| 458 | |
| 459 | 1. Call ``CPU_ON`` and ``CPU_OFF`` on each non-lead CPU in sequence. |
| 460 | |
| 461 | 2. Program wake up timer and suspend the lead CPU to the deepest power level. |
| 462 | |
| 463 | 3. Call ``CPU_ON`` on non-lead CPU to get the timestamps from each CPU. |
| 464 | |
| 465 | +-------+---------------------+--------------------+--------------------------+ |
| 466 | | CPU | ``PSCI_ENTRY`` (us) | ``PSCI_EXIT`` (us) | ``CFLUSH_OVERHEAD`` (us) | |
| 467 | +=======+=====================+====================+==========================+ |
| 468 | | 0 | 110 | 28 | 93 | |
| 469 | +-------+---------------------+--------------------+--------------------------+ |
| 470 | | 1 | 110 | 28 | 93 | |
| 471 | +-------+---------------------+--------------------+--------------------------+ |
| 472 | | 2 | 110 | 28 | 93 | |
| 473 | +-------+---------------------+--------------------+--------------------------+ |
| 474 | | 3 | 111 | 28 | 93 | |
| 475 | +-------+---------------------+--------------------+--------------------------+ |
| 476 | | 4 | 195 | 22 | 181 | |
| 477 | +-------+---------------------+--------------------+--------------------------+ |
| 478 | | 5 | 20 | 23 | 6 | |
| 479 | +-------+---------------------+--------------------+--------------------------+ |
| 480 | |
| 481 | The ``CFLUSH_OVERHEAD`` times for all little CPUs are large because all other |
| 482 | CPUs in that cluster are powerered down during the test. The ``CPU_OFF`` call |
| 483 | powers down to the cluster level, requiring a flush of both L1 and L2 caches. |
| 484 | |
| 485 | The ``PSCI_ENTRY`` and ``CFLUSH_OVERHEAD`` times for CPU 5 are small because |
| 486 | lead CPU 4 is running and CPU 5 only powers down to level 0, which only requires |
| 487 | an L1 cache flush. |
| 488 | |
| 489 | The ``CFLUSH_OVERHEAD`` time for CPU 4 is a lot larger than those for the little |
| 490 | CPUs because the L2 cache size for the big cluster is lot larger (2MB) compared |
| 491 | to the little cluster (1MB). |
| 492 | |
| 493 | The ``PSCI_EXIT`` times for CPUs in the big cluster are slightly smaller than |
| 494 | for CPUs in the little cluster due to greater CPU performance. These times |
| 495 | generally are greater than the ``PSCI_EXIT`` times in the ``CPU_SUSPEND`` tests |
| 496 | because there is more code to execute in the "on finisher" compared to the |
| 497 | "suspend finisher" (for example, GIC redistributor register programming). |
| 498 | |
| 499 | ``PSCI_VERSION`` on all CPUs in parallel |
| 500 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 501 | |
| 502 | Since very little code is associated with ``PSCI_VERSION``, this test |
| 503 | approximates the round trip latency for handling a fast SMC at EL3 in TF. |
| 504 | |
| 505 | +-------+-------------------+ |
| 506 | | CPU | TOTAL TIME (ns) | |
| 507 | +=======+===================+ |
| 508 | | 0 | 3020 | |
| 509 | +-------+-------------------+ |
| 510 | | 1 | 2940 | |
| 511 | +-------+-------------------+ |
| 512 | | 2 | 2980 | |
| 513 | +-------+-------------------+ |
| 514 | | 3 | 3060 | |
| 515 | +-------+-------------------+ |
| 516 | | 4 | 520 | |
| 517 | +-------+-------------------+ |
| 518 | | 5 | 720 | |
| 519 | +-------+-------------------+ |
| 520 | |
| 521 | The times for the big CPUs are less than the little CPUs due to greater CPU |
| 522 | performance. |
| 523 | |
| 524 | We suspect the time for lead CPU 4 is shorter than CPU 5 due to subtle cache |
| 525 | effects, given that these measurements are at the nano-second level. |
| 526 | |
John Tsichritzis | 63801cd | 2019-07-05 14:22:12 +0100 | [diff] [blame] | 527 | -------------- |
| 528 | |
Harrison Mutai | 341740c | 2023-02-13 18:30:04 +0000 | [diff] [blame] | 529 | *Copyright (c) 2019-2023, Arm Limited and Contributors. All rights reserved.* |
John Tsichritzis | 63801cd | 2019-07-05 14:22:12 +0100 | [diff] [blame] | 530 | |
Harrison Mutai | 341740c | 2023-02-13 18:30:04 +0000 | [diff] [blame] | 531 | .. _Juno R1 platform: https://developer.arm.com/documentation/100122/latest/ |
Joel Hutton | 0f79fb1 | 2019-02-26 16:23:54 +0000 | [diff] [blame] | 532 | .. _TF master as of 31/01/2017: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/?id=c38b36d |
Harrison Mutai | 21cb965 | 2023-05-17 13:09:16 +0100 | [diff] [blame] | 533 | .. _v2.9-rc0: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/?h=v2.9-rc0 |