blob: 92fbd22721942898705160b82f865ed38f1507a8 [file] [log] [blame]
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +01001U-Boot pytest suite
2===================
Stephen Warren10e50632016-01-15 11:15:24 -07003
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +01004Introduction
5------------
Stephen Warren10e50632016-01-15 11:15:24 -07006
7This tool aims to test U-Boot by executing U-Boot shell commands using the
8console interface. A single top-level script exists to execute or attach to the
9U-Boot console, run the entire script of tests against it, and summarize the
10results. Advantages of this approach are:
11
12- Testing is performed in the same way a user or script would interact with
13 U-Boot; there can be no disconnect.
14- There is no need to write or embed test-related code into U-Boot itself.
15 It is asserted that writing test-related code in Python is simpler and more
Simon Glass25404102021-03-07 17:35:17 -070016 flexible than writing it all in C. But see :doc:`tests_writing` for caveats
17 and more discussion / analysis.
Stephen Warren10e50632016-01-15 11:15:24 -070018- It is reasonably simple to interact with U-Boot in this way.
19
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010020Requirements
21------------
Stephen Warren10e50632016-01-15 11:15:24 -070022
23The test suite is implemented using pytest. Interaction with the U-Boot console
24involves executing some binary and interacting with its stdin/stdout. You will
25need to implement various "hook" scripts that are called by the test suite at
26the appropriate time.
27
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010028In order to run the test suite at a minimum we require that both Python 3 and
29pip for Python 3 are installed. All of the required python modules are
30described in the requirements.txt file in the /test/py/ directory and can be
31installed via the command
Stephen Warren10e50632016-01-15 11:15:24 -070032
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010033.. code-block:: bash
Tom Rinia39e81b2019-10-24 11:59:26 -040034
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010035 pip install -r requirements.txt
36
37In order to execute certain tests on their supported platforms other tools
38will be required. The following is an incomplete list:
AKASHI Takahiro58c95c22020-04-14 11:51:49 +090039
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010040* gdisk
41* dfu-util
42* dtc
43* openssl
44* sudo OR guestmount
45* e2fsprogs
46* util-linux
47* coreutils
48* dosfstools
49* efitools
50* mount
51* mtools
52* sbsigntool
53* udisks2
Tom Rinia39e81b2019-10-24 11:59:26 -040054
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010055Please use the appropriate commands for your distribution to match these tools
Tom Rinia39e81b2019-10-24 11:59:26 -040056up with the package that provides them.
Stephen Warren10e50632016-01-15 11:15:24 -070057
58The test script supports either:
59
60- Executing a sandbox port of U-Boot on the local machine as a sub-process,
61 and interacting with it over stdin/stdout.
62- Executing an external "hook" scripts to flash a U-Boot binary onto a
63 physical board, attach to the board's console stream, and reset the board.
64 Further details are described later.
65
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010066Using `virtualenv` to provide requirements
67~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -070068
Tom Rinia39e81b2019-10-24 11:59:26 -040069The recommended way to run the test suite, in order to ensure reproducibility
70is to use `virtualenv` to set up the necessary environment. This can be done
71via the following commands:
Stephen Warren10e50632016-01-15 11:15:24 -070072
Stephen Warren10e50632016-01-15 11:15:24 -070073
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010074.. code-block:: console
Stephen Warren10e50632016-01-15 11:15:24 -070075
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010076 $ cd /path/to/u-boot
77 $ sudo apt-get install python3 python3-virtualenv
78 $ virtualenv -p /usr/bin/python3 venv
79 $ . ./venv/bin/activate
80 $ pip install -r test/py/requirements.txt
81
82Testing sandbox
83---------------
84
85To run the test suite on the sandbox port (U-Boot built as a native user-space
Stephen Warren10e50632016-01-15 11:15:24 -070086application), simply execute:
87
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +010088.. code-block:: bash
89
90 ./test/py/test.py --bd sandbox --build
Stephen Warren10e50632016-01-15 11:15:24 -070091
92The `--bd` option tells the test suite which board type is being tested. This
93lets the test suite know which features the board has, and hence exactly what
94can be tested.
95
96The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
97omit this option and build U-Boot yourself, in whatever way you choose, before
98running the test script.
99
100The test script will attach to U-Boot, execute all valid tests for the board,
101then print a summary of the test process. A complete log of the test session
102will be written to `${build_dir}/test-log.html`. This is best viewed in a web
103browser, but may be read directly as plain text, perhaps with the aid of the
104`html2text` utility.
105
Simon Glassafd00042021-10-08 09:15:23 -0600106If sandbox crashes (e.g. with a segfault) you will see message like this::
107
108
109 test/py/u_boot_spawn.py:171: in expect
110 c = os.read(self.fd, 1024).decode(errors='replace')
111 E ValueError: U-Boot exited with signal 11 (Signals.SIGSEGV)
112
113
Simon Glass19193482021-10-05 20:18:00 -0600114Controlling output
115~~~~~~~~~~~~~~~~~~
116
117By default a short backtrace is reported. If you would like a longer one,
118pass ``--tb=long`` when running the test. See the pytest documentation for
119more options.
120
Simon Glass5c8b8842021-09-19 15:14:51 -0600121Running tests in parallel
122~~~~~~~~~~~~~~~~~~~~~~~~~
123
Simon Glassb23b98f2022-08-06 17:51:59 -0600124Note: Not all tests can run in parallel at present, so the usual approach is
125to just run those that can.
Simon Glass5c8b8842021-09-19 15:14:51 -0600126
127First install support for parallel tests::
128
Simon Glassb23b98f2022-08-06 17:51:59 -0600129 sudo apt install python3-pytest-xdist
130
131or:::
132
Simon Glass5c8b8842021-09-19 15:14:51 -0600133 pip3 install pytest-xdist
134
Simon Glassb23b98f2022-08-06 17:51:59 -0600135Then run the tests in parallel using the -n flag::
Simon Glass5c8b8842021-09-19 15:14:51 -0600136
Simon Glassb23b98f2022-08-06 17:51:59 -0600137 test/py/test.py -B sandbox --build --build-dir /tmp/b/sandbox -q -k \
138 'not slow and not bootstd and not spi_flash' -n16
Simon Glass5c8b8842021-09-19 15:14:51 -0600139
Simon Glassb23b98f2022-08-06 17:51:59 -0600140You can also use `make pcheck` to run all tests in parallel. This uses a maximum
141of 16 threads, since the setup time is significant and there are under 1000
142tests.
Simon Glass5c8b8842021-09-19 15:14:51 -0600143
Simon Glassb23b98f2022-08-06 17:51:59 -0600144Note that the `test-log.html` output does not work correctly at present with
145parallel testing. All the threads write to it at once, so it is garbled.
Simon Glass5c8b8842021-09-19 15:14:51 -0600146
Simon Glassb23b98f2022-08-06 17:51:59 -0600147Note that the `tools/` tests still run each tool's tests once after the other,
148although within that, they do run in parallel. So for example, the buildman
149tests run in parallel, then the binman tests run in parallel. There would be a
150significant advantage to running them all in parallel together, but that would
151require a large amount of refactoring, e.g. with more use of pytest fixtures.
152The code-coverage tests are omitted since they cannot run in parallel due to a
153Python limitation.
Simon Glass5c8b8842021-09-19 15:14:51 -0600154
155
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100156Testing under a debugger
157~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warrenc9afc502016-02-08 14:49:02 -0700158
159If you need to run sandbox under a debugger, you may pass the command-line
160option `--gdbserver COMM`. This causes two things to happens:
161
162- Instead of running U-Boot directly, it will be run under gdbserver, with
163 debug communication via the channel `COMM`. You can attach a debugger to the
164 sandbox process in order to debug it. See `man gdbserver` and the example
165 below for details of valid values for `COMM`.
166- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
167 time to execute commands. This is useful if U-Boot is stopped at a breakpoint
168 during debugging.
169
170A usage example is:
171
172Window 1:
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100173
174.. code-block:: bash
175
176 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
Stephen Warrenc9afc502016-02-08 14:49:02 -0700177
178Window 2:
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100179
180.. code-block:: bash
181
182 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
Stephen Warrenc9afc502016-02-08 14:49:02 -0700183
184Alternatively, you could leave off the `-ex` option and type the command
185manually into gdb once it starts.
186
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100187You can use any debugger you wish, as long as it speaks the gdb remote
Stephen Warrenc9afc502016-02-08 14:49:02 -0700188protocol, or any graphical wrapper around gdb.
189
190Some tests deliberately cause the sandbox process to exit, e.g. to test the
191reset command, or sandbox's CTRL-C handling. When this happens, you will need
192to attach the debugger to the new sandbox instance. If these tests are not
193relevant to your debugging session, you can skip them using pytest's -k
194command-line option; see the next section.
195
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100196Command-line options
197--------------------
198
199--board-type, --bd, -B
200 set the type of the board to be tested. For example, `sandbox` or `seaboard`.
Stephen Warren10e50632016-01-15 11:15:24 -0700201
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100202--board-identity`, --id
203 sets the identity of the board to be tested. This allows differentiation
204 between multiple instances of the same type of physical board that are
205 attached to the same host machine. This parameter is not interpreted by th
206 test script in any way, but rather is simply passed to the hook scripts
207 described below, and may be used in any site-specific way deemed necessary.
208
209--build
210 indicates that the test script should compile U-Boot itself before running
211 the tests. If using this option, make sure that any environment variables
212 required by the build process are already set, such as `$CROSS_COMPILE`.
213
214--buildman
215 indicates that `--build` should use buildman to build U-Boot. There is no need
216 to set $CROSS_COMPILE` in this case since buildman handles it.
217
218--build-dir
219 sets the directory containing the compiled U-Boot binaries. If omitted, this
220 is `${source_dir}/build-${board_type}`.
221
222--result-dir
223 sets the directory to write results, such as log files, into.
224 If omitted, the build directory is used.
225
226--persistent-data-dir
227 sets the directory used to store persistent test data. This is test data that
228 may be re-used across test runs, such as file-system images.
Stephen Warren10e50632016-01-15 11:15:24 -0700229
Stephen Warrenc9afc502016-02-08 14:49:02 -0700230`pytest` also implements a number of its own command-line options. Commonly used
231options are mentioned below. Please see `pytest` documentation for complete
232details. Execute `py.test --version` for a brief summary. Note that U-Boot's
233test.py script passes all command-line arguments directly to `pytest` for
234processing.
235
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100236-k
237 selects which tests to run. The default is to run all known tests. This
Stephen Warrenc9afc502016-02-08 14:49:02 -0700238 option takes a single argument which is used to filter test names. Simple
239 logical operators are supported. For example:
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100240
241 - `'-k ums'` runs only tests with "ums" in their name.
242 - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
Stephen Warrenc9afc502016-02-08 14:49:02 -0700243 case, "ut_dm" is a parameter to a test rather than the test name. The full
244 test name is e.g. "test_ut[ut_dm_leak]".
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100245 - `'-k not reset'` runs everything except tests with "reset" in their name.
246 - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
247 - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
Stephen Warrenc9afc502016-02-08 14:49:02 -0700248 their name.
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100249
250-s
251 prevents pytest from hiding a test's stdout. This allows you to see
Stephen Warrenc9afc502016-02-08 14:49:02 -0700252 U-Boot's console log in real time on pytest's stdout.
Stephen Warren10e50632016-01-15 11:15:24 -0700253
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100254Testing real hardware
255---------------------
Stephen Warren10e50632016-01-15 11:15:24 -0700256
257The tools and techniques used to interact with real hardware will vary
258radically between different host and target systems, and the whims of the user.
259For this reason, the test suite does not attempt to directly interact with real
260hardware in any way. Rather, it executes a standardized set of "hook" scripts
261via `$PATH`. These scripts implement certain actions on behalf of the test
262suite. This keeps the test suite simple and isolated from system variances
263unrelated to U-Boot features.
264
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100265Hook scripts
266~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700267
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100268Environment variables
269'''''''''''''''''''''
Stephen Warren10e50632016-01-15 11:15:24 -0700270
271The following environment variables are set when running hook scripts:
272
273- `UBOOT_BOARD_TYPE` the board type being tested.
274- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
275 specified.
276- `UBOOT_SOURCE_DIR` the U-Boot source directory.
277- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
278- `UBOOT_BUILD_DIR` the U-Boot build directory.
279- `UBOOT_RESULT_DIR` the test result directory.
Masahiro Yamadac9018912017-10-19 19:08:52 +0900280- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
Stephen Warren10e50632016-01-15 11:15:24 -0700281
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100282u-boot-test-console
283'''''''''''''''''''
Stephen Warren10e50632016-01-15 11:15:24 -0700284
285This script provides access to the U-Boot console. The script's stdin/stdout
286should be connected to the board's console. This process should continue to run
287indefinitely, until killed. The test suite will run this script in parallel
288with all other hooks.
289
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100290This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
291via exec().
Stephen Warren10e50632016-01-15 11:15:24 -0700292
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100293If you are able to run U-Boot under a hardware simulator such as QEMU, then
Stephen Warren10e50632016-01-15 11:15:24 -0700294you would likely spawn that simulator from this script. However, note that
295`u-boot-test-reset` may be called multiple times per test script run, and must
296cause U-Boot to start execution from scratch each time. Hopefully your
297simulator includes a virtual reset button! If not, you can launch the
298simulator from `u-boot-test-reset` instead, while arranging for this console
299process to always communicate with the current simulator instance.
300
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100301u-boot-test-flash
302'''''''''''''''''
Stephen Warren10e50632016-01-15 11:15:24 -0700303
304Prior to running the test suite against a board, some arrangement must be made
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100305so that the board executes the particular U-Boot binary to be tested. Often
Stephen Warren10e50632016-01-15 11:15:24 -0700306this involves writing the U-Boot binary to the board's flash ROM. The test
307suite calls this hook script for that purpose.
308
309This script should perform the entire flashing process synchronously; the
310script should only exit once flashing is complete, and a board reset will
311cause the newly flashed U-Boot binary to be executed.
312
313It is conceivable that this script will do nothing. This might be useful in
314the following cases:
315
316- Some other process has already written the desired U-Boot binary into the
317 board's flash prior to running the test suite.
318- The board allows U-Boot to be downloaded directly into RAM, and executed
319 from there. Use of this feature will reduce wear on the board's flash, so
320 may be preferable if available, and if cold boot testing of U-Boot is not
321 required. If this feature is used, the `u-boot-test-reset` script should
Masahiro Yamadac9018912017-10-19 19:08:52 +0900322 perform this download, since the board could conceivably be reset multiple
Stephen Warren10e50632016-01-15 11:15:24 -0700323 times in a single test run.
324
325It is up to the user to determine if those situations exist, and to code this
326hook script appropriately.
327
328This script will typically be implemented by calling out to some SoC- or
329board-specific vendor flashing utility.
330
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100331u-boot-test-reset
332'''''''''''''''''
Stephen Warren10e50632016-01-15 11:15:24 -0700333
334Whenever the test suite needs to reset the target board, this script is
335executed. This is guaranteed to happen at least once, prior to executing the
336first test function. If any test fails, the test infra-structure will execute
337this script again to restore U-Boot to an operational state before running the
338next test function.
339
340This script will likely be implemented by communicating with some form of
341relay or electronic switch attached to the board's reset signal.
342
343The semantics of this script require that when it is executed, U-Boot will
344start running from scratch. If the U-Boot binary to be tested has been written
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100345to flash, pulsing the board's reset signal is likely all this script needs to
346do. However, in some scenarios, this script may perform other actions. For
Stephen Warren10e50632016-01-15 11:15:24 -0700347example, it may call out to some SoC- or board-specific vendor utility in order
348to download the U-Boot binary directly into RAM and execute it. This would
349avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
350saving wear on the flash chip(s).
351
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100352Examples
353''''''''
Stephen Warren140d04d2016-04-06 11:46:59 -0600354
Tom Rini2a1df582021-02-24 17:05:04 -0500355https://source.denx.de/u-boot/u-boot-test-hooks contains some working example hook
Stephen Warren140d04d2016-04-06 11:46:59 -0600356scripts, and may be useful as a reference when implementing hook scripts for
357your platform. These scripts are not considered part of U-Boot itself.
358
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100359Board-type-specific configuration
360~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700361
362Each board has a different configuration and behaviour. Many of these
363differences can be automatically detected by parsing the `.config` file in the
364build directory. However, some differences can't yet be handled automatically.
365
366For each board, an optional Python module `u_boot_board_${board_type}` may exist
367to provide board-specific information to the test script. Any global value
368defined in these modules is available for use by any test function. The data
369contained in these scripts must be purely derived from U-Boot source code.
370Hence, these configuration files are part of the U-Boot source tree too.
371
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100372Execution environment configuration
373~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700374
375Each user's hardware setup may enable testing different subsets of the features
376implemented by a particular board's configuration of U-Boot. For example, a
377U-Boot configuration may support USB device mode and USB Mass Storage, but this
378can only be tested if a USB cable is connected between the board and the host
379machine running the test script.
380
381For each board, optional Python modules `u_boot_boardenv_${board_type}` and
382`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
383board-specific and board-identity-specific information to the test script. Any
384global value defined in these modules is available for use by any test
385function. The data contained in these is specific to a particular user's
386hardware configuration. Hence, these configuration files are not part of the
387U-Boot source tree, and should be installed outside of the source tree. Users
388should set `$PYTHONPATH` prior to running the test script to allow these
389modules to be loaded.
390
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100391Board module parameter usage
392~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700393
394The test scripts rely on the following variables being defined by the board
395module:
396
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100397- none at present
Stephen Warren10e50632016-01-15 11:15:24 -0700398
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100399U-Boot `.config` feature usage
400~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700401
402The test scripts rely on various U-Boot `.config` features, either directly in
403order to test those features, or indirectly in order to query information from
404the running U-Boot instance in order to test other features.
405
406One example is that testing of the `md` command requires knowledge of a RAM
407address to use for the test. This data is parsed from the output of the
408`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
409
410For a complete list of dependencies, please search the test scripts for
411instances of:
412
413- `buildconfig.get(...`
414- `@pytest.mark.buildconfigspec(...`
Heinrich Schuchardt59264ae2019-04-22 09:18:55 +0200415- `@pytest.mark.notbuildconfigspec(...`
Stephen Warren10e50632016-01-15 11:15:24 -0700416
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100417Complete invocation example
418~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stephen Warren10e50632016-01-15 11:15:24 -0700419
420Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
421any required environment configuration Python modules into $HOME/ubtest/py,
422then you would likely invoke the test script as follows:
423
424If U-Boot has already been built:
425
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100426.. code-block:: bash
427
428 PATH=$HOME/ubtest/bin:$PATH \
Liam Beguince3f90e2018-03-14 19:15:13 -0400429 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
Stephen Warren10e50632016-01-15 11:15:24 -0700430 ./test/py/test.py --bd seaboard
Stephen Warren10e50632016-01-15 11:15:24 -0700431
432If you want the test script to compile U-Boot for you too, then you likely
433need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
Simon Glass6e094842020-03-18 09:43:01 -0600434follows:
Stephen Warren10e50632016-01-15 11:15:24 -0700435
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100436.. code-block:: bash
437
438 CROSS_COMPILE=arm-none-eabi- \
Stephen Warren10e50632016-01-15 11:15:24 -0700439 PATH=$HOME/ubtest/bin:$PATH \
Liam Beguince3f90e2018-03-14 19:15:13 -0400440 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
Stephen Warren10e50632016-01-15 11:15:24 -0700441 ./test/py/test.py --bd seaboard --build
Stephen Warren10e50632016-01-15 11:15:24 -0700442
Simon Glass6e094842020-03-18 09:43:01 -0600443or, using buildman to handle it:
444
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100445.. code-block:: bash
446
Simon Glass6e094842020-03-18 09:43:01 -0600447 PATH=$HOME/ubtest/bin:$PATH \
448 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
449 ./test/py/test.py --bd seaboard --build --buildman
Simon Glass6e094842020-03-18 09:43:01 -0600450
Heinrich Schuchardt79c9f0e2021-01-18 20:24:03 +0100451Writing tests
452-------------
Stephen Warren10e50632016-01-15 11:15:24 -0700453
454Please refer to the pytest documentation for details of writing pytest tests.
455Details specific to the U-Boot test suite are described below.
456
457A test fixture named `u_boot_console` should be used by each test function. This
458provides the means to interact with the U-Boot console, and retrieve board and
459environment configuration information.
460
461The function `u_boot_console.run_command()` executes a shell command on the
462U-Boot console, and returns all output from that command. This allows
463validation or interpretation of the command output. This function validates
464that certain strings are not seen on the U-Boot console. These include shell
465error messages and the U-Boot sign-on message (in order to detect unexpected
466board resets). See the source of `u_boot_console_base.py` for a complete list of
467"bad" strings. Some test scenarios are expected to trigger these strings. Use
468`u_boot_console.disable_check()` to temporarily disable checking for specific
469strings. See `test_unknown_cmd.py` for an example.
470
471Board- and board-environment configuration values may be accessed as sub-fields
472of the `u_boot_console.config` object, for example
473`u_boot_console.config.ram_base`.
474
475Build configuration values (from `.config`) may be accessed via the dictionary
476`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
477names.