Simon Glass | c3730d7 | 2021-07-21 21:35:49 -0600 | [diff] [blame] | 1 | .. SPDX-License-Identifier: GPL-2.0+ |
| 2 | |
| 3 | Introduction to testing |
| 4 | ======================= |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 5 | |
| 6 | U-Boot has a large amount of code. This file describes how this code is |
| 7 | tested and what tests you should write when adding a new feature. |
| 8 | |
| 9 | |
Simon Glass | 8ab6cc3 | 2016-07-03 09:40:34 -0600 | [diff] [blame] | 10 | Running tests |
| 11 | ------------- |
| 12 | |
Simon Glass | cb605d2 | 2021-03-07 17:34:38 -0700 | [diff] [blame] | 13 | To run most tests on sandbox, type this:: |
Simon Glass | 8ab6cc3 | 2016-07-03 09:40:34 -0600 | [diff] [blame] | 14 | |
Simon Glass | 3fcb828 | 2018-11-18 08:14:29 -0700 | [diff] [blame] | 15 | make check |
Simon Glass | 8ab6cc3 | 2016-07-03 09:40:34 -0600 | [diff] [blame] | 16 | |
| 17 | in the U-Boot directory. Note that only the pytest suite is run using this |
Simon Glass | dc3ab7e | 2016-07-31 17:35:02 -0600 | [diff] [blame] | 18 | command. |
Simon Glass | 8ab6cc3 | 2016-07-03 09:40:34 -0600 | [diff] [blame] | 19 | |
Simon Glass | 3602c55 | 2021-03-07 17:34:39 -0700 | [diff] [blame] | 20 | Some tests take ages to run and are marked with @pytest.mark.slow. To run just |
| 21 | the quick ones, type this:: |
Simon Glass | 3fcb828 | 2018-11-18 08:14:29 -0700 | [diff] [blame] | 22 | |
| 23 | make qcheck |
| 24 | |
Simon Glass | 3602c55 | 2021-03-07 17:34:39 -0700 | [diff] [blame] | 25 | It is also possible to run just the tests for tools (patman, binman, etc.). |
| 26 | Such tests are included with those tools, i.e. no actual U-Boot unit tests are |
| 27 | run. Type this:: |
| 28 | |
| 29 | make tcheck |
| 30 | |
| 31 | All of the above use the test/run script with a paremeter to select which tests |
| 32 | are run. |
| 33 | |
Simon Glass | 8ab6cc3 | 2016-07-03 09:40:34 -0600 | [diff] [blame] | 34 | |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 35 | Sandbox |
| 36 | ------- |
| 37 | U-Boot can be built as a user-space application (e.g. for Linux). This |
| 38 | allows test to be executed without needing target hardware. The 'sandbox' |
| 39 | target provides this feature and it is widely used in tests. |
| 40 | |
Simon Glass | 9e78ad6 | 2021-03-07 17:34:42 -0700 | [diff] [blame] | 41 | See :doc:`tests_sandbox` for more information. |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 42 | |
| 43 | Pytest Suite |
| 44 | ------------ |
| 45 | |
| 46 | Many tests are available using the pytest suite, in test/py. This can run |
| 47 | either on sandbox or on real hardware. It relies on the U-Boot console to |
| 48 | inject test commands and check the result. It is slower to run than C code, |
Simon Glass | dc3ab7e | 2016-07-31 17:35:02 -0600 | [diff] [blame] | 49 | but provides the ability to unify lots of tests and summarise their results. |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 50 | |
Simon Glass | cb605d2 | 2021-03-07 17:34:38 -0700 | [diff] [blame] | 51 | You can run the tests on sandbox with:: |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 52 | |
Simon Glass | cb605d2 | 2021-03-07 17:34:38 -0700 | [diff] [blame] | 53 | ./test/py/test.py --bd sandbox --build |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 54 | |
| 55 | This will produce HTML output in build-sandbox/test-log.html |
| 56 | |
Simon Glass | 9e78ad6 | 2021-03-07 17:34:42 -0700 | [diff] [blame] | 57 | Some tests run with other versions of sandbox. For example sandbox_flattree |
| 58 | runs the tests with livetree (the hierachical devicetree) disabled. You can |
| 59 | also select particular tests with -k:: |
| 60 | |
| 61 | ./test/py/test.py --bd sandbox_flattree --build -k hello |
| 62 | |
Simon Glass | eaf2196 | 2021-03-07 17:34:43 -0700 | [diff] [blame] | 63 | There are some special tests that run in SPL. For this you need the sandbox_spl |
| 64 | build:: |
| 65 | |
| 66 | ./test/py/test.py --bd sandbox_spl --build -k test_spl |
| 67 | |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 68 | See test/py/README.md for more information about the pytest suite. |
| 69 | |
Simon Glass | 9e78ad6 | 2021-03-07 17:34:42 -0700 | [diff] [blame] | 70 | See :doc:`tests_sandbox` for how to run tests directly (not through pytest). |
| 71 | |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 72 | |
| 73 | tbot |
| 74 | ---- |
| 75 | |
| 76 | Tbot provides a way to execute tests on target hardware. It is intended for |
| 77 | trying out both U-Boot and Linux (and potentially other software) on a |
| 78 | number of boards automatically. It can be used to create a continuous test |
Heiko Schocher | d551b84 | 2017-06-09 06:13:34 +0200 | [diff] [blame] | 79 | environment. See http://www.tbot.tools for more information. |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 80 | |
| 81 | |
| 82 | Ad-hoc tests |
| 83 | ------------ |
| 84 | |
| 85 | There are several ad-hoc tests which run outside the pytest environment: |
| 86 | |
Simon Glass | cb605d2 | 2021-03-07 17:34:38 -0700 | [diff] [blame] | 87 | test/fs |
| 88 | File system test (shell script) |
| 89 | test/image |
| 90 | FIT and legacy image tests (shell script and Python) |
| 91 | test/stdint |
| 92 | A test that stdint.h can be used in U-Boot (shell script) |
| 93 | trace |
| 94 | Test for the tracing feature (shell script) |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 95 | |
Simon Glass | dc3ab7e | 2016-07-31 17:35:02 -0600 | [diff] [blame] | 96 | TODO: Move these into pytest. |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 97 | |
| 98 | |
| 99 | When to write tests |
| 100 | ------------------- |
| 101 | |
| 102 | If you add code to U-Boot without a test you are taking a risk. Even if you |
| 103 | perform thorough manual testing at the time of submission, it may break when |
| 104 | future changes are made to U-Boot. It may even break when applied to mainline, |
| 105 | if other changes interact with it. A good mindset is that untested code |
| 106 | probably doesn't work and should be deleted. |
| 107 | |
| 108 | You can assume that the Pytest suite will be run before patches are accepted |
| 109 | to mainline, so this provides protection against future breakage. |
| 110 | |
| 111 | On the other hand there is quite a bit of code that is not covered with tests, |
| 112 | or is covered sparingly. So here are some suggestions: |
| 113 | |
| 114 | - If you are adding a new uclass, add a sandbox driver and a test that uses it |
| 115 | - If you are modifying code covered by an existing test, add a new test case |
| 116 | to cover your changes |
| 117 | - If the code you are modifying has not tests, consider writing one. Even a |
| 118 | very basic test is useful, and may be picked up and enhanced by others. It |
| 119 | is much easier to add onto a test - writing a new large test can seem |
| 120 | daunting to most contributors. |
| 121 | |
Simon Glass | 2540410 | 2021-03-07 17:35:17 -0700 | [diff] [blame] | 122 | See doc:`tests_writing` for how to write tests. |
| 123 | |
Simon Glass | d8e3635 | 2016-07-03 09:40:33 -0600 | [diff] [blame] | 124 | |
| 125 | Future work |
| 126 | ----------- |
| 127 | |
| 128 | Converting existing shell scripts into pytest tests. |