test: Support tests which can only be run manually
At present we normally write tests either in Python or in C. But most
Python tests end up doing a lot of checks which would be better done in C.
Checks done in C are orders of magnitude faster and it is possible to get
full access to U-Boot's internal workings, rather than just relying on
the command line.
The model is to have a Python test set up some things and then use C code
(in a unit test) to check that they were done correctly. But we don't want
those checks to happen as part of normal test running, since each C unit
tests is dependent on the associate Python tests, so cannot run without
it.
To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
tests, so that they can be skipped by default when the 'ut' command is
used. Require that tests have a name ending with '_norun', so that pytest
knows to skip them.
Signed-off-by: Simon Glass <sjg@chromium.org>
diff --git a/test/test-main.c b/test/test-main.c
index 2323cba..ddfd89c 100644
--- a/test/test-main.c
+++ b/test/test-main.c
@@ -508,6 +508,30 @@
if (!test_matches(prefix, test_name, select_name))
continue;
+
+ if (test->flags & UT_TESTF_MANUAL) {
+ int len;
+
+ /*
+ * manual tests must have a name ending "_norun" as this
+ * is how pytest knows to skip them. See
+ * generate_ut_subtest() for this check.
+ */
+ len = strlen(test_name);
+ if (len < 6 || strcmp(test_name + len - 6, "_norun")) {
+ printf("Test %s is manual so must have a name ending in _norun\n",
+ test_name);
+ uts->fail_count++;
+ return -EBADF;
+ }
+ if (!uts->force_run) {
+ if (select_name) {
+ printf("Test %s skipped as it is manual (use -f to run it)\n",
+ test_name);
+ }
+ continue;
+ }
+ }
old_fail_count = uts->fail_count;
for (i = 0; i < uts->runs_per_test; i++)
ret = ut_run_test_live_flat(uts, test, select_name);
@@ -529,7 +553,7 @@
int ut_run_list(const char *category, const char *prefix,
struct unit_test *tests, int count, const char *select_name,
- int runs_per_test)
+ int runs_per_test, bool force_run)
{
struct unit_test_state uts = { .fail_count = 0 };
bool has_dm_tests = false;
@@ -563,6 +587,7 @@
}
memcpy(uts.fdt_copy, gd->fdt_blob, uts.fdt_size);
}
+ uts.force_run = force_run;
ret = ut_run_tests(&uts, prefix, tests, count, select_name);
/* Best efforts only...ignore errors */