Merge patch series "labgrid: Provide an integration with Labgrid"
Simon Glass <sjg@chromium.org> says:
Labgrid provides access to a hardware lab in an automated way. It is
possible to boot U-Boot on boards in the lab without physically touching
them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1]
which allowed my lab of about 30 devices to be operated remotely, using
tbot for the console and build integration. While it worked OK and I
used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid,
which I did not know about at time (thank you Tom for telling me). It is
more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be
remedied.
On and off over the past several weeks I have been experimenting with
Labgrid. I have managed to create an initial U-Boot integration (this
series) by adding various features to Labgrid[2] and the U-Boot test
hooks.
I hope that this might inspire others to set up boards and run tests
automatically, rather than relying on infrequent, manual test. Perhaps
it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it
easy to connect to boards and run tests:
ub-int <target>
Build and boot on a target, starting an interactive session
ub-cli <target>
Build and boot on a target, ensure U-Boot starts and provide an interactive
session from there
ub-smoke <target>
Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target>
Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec>
Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have
relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect
patches from the mailing list, apply them and test them on a selection
of boards. I suspect that script already exists, so please let me know
what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a
[2] https://github.com/labgrid-project/labgrid/pull/1411
[3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid
[4] https://tbot.tools/index.html
Link: https://lore.kernel.org/r/20241112141326.643128-1-sjg@chromium.org
[trini: Move the sjg-lab job to prior to world build, to fix pipeline
status]
Signed-off-by: Tom Rini <trini@konsulko.com>
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0aeda53..2164ad7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -3,6 +3,7 @@
variables:
DEFAULT_TAG: ""
MIRROR_DOCKER: docker.io
+ SJG_LAB: ""
default:
tags:
@@ -16,6 +17,7 @@
stages:
- testsuites
- test.py
+ - sjg-lab
- world build
.buildman_and_testpy_template: &buildman_and_testpy_dfn
@@ -521,3 +523,158 @@
TEST_PY_TEST_SPEC: "not sleep"
TEST_PY_ID: "--id qemu"
<<: *buildman_and_testpy_dfn
+
+.lab_template: &lab_dfn
+ stage: sjg-lab
+ rules:
+ - if: $SJG_LAB == "1"
+ when: always
+ - if: $SJG_LAB != "1"
+ when: manual
+ allow_failure: true
+ tags: [ 'lab' ]
+ script:
+ - if [[ -z "${SJG_LAB}" ]]; then
+ exit 0;
+ fi
+ # Environment:
+ # SRC - source tree
+ # OUT - output directory for builds
+ - export SRC="$(pwd)"
+ - export OUT="${SRC}/build/${BOARD}"
+ - export PATH=$PATH:~/bin
+ - export PATH=$PATH:/vid/software/devel/ubtest/u-boot-test-hooks/bin
+
+ # Load it on the device
+ - ret=0
+ - echo "role ${ROLE}"
+ - export strategy="-s uboot -e off"
+ - export USE_LABGRID_SJG=1
+ # export verbose="-v"
+ - ${SRC}/test/py/test.py --role ${ROLE} --build-dir "${OUT}"
+ --capture=tee-sys -k "not bootstd" || ret=$?
+ - U_BOOT_BOARD_IDENTITY="${ROLE}" u-boot-test-release || true
+ - if [[ $ret -ne 0 ]]; then
+ exit $ret;
+ fi
+ artifacts:
+ when: always
+ paths:
+ - "build/${BOARD}/test-log.html"
+ - "build/${BOARD}/multiplexed_log.css"
+ expire_in: 1 week
+
+rpi3:
+ variables:
+ ROLE: rpi3
+ <<: *lab_dfn
+
+opi_pc:
+ variables:
+ ROLE: opi_pc
+ <<: *lab_dfn
+
+pcduino3_nano:
+ variables:
+ ROLE: pcduino3_nano
+ <<: *lab_dfn
+
+samus:
+ variables:
+ ROLE: samus
+ <<: *lab_dfn
+
+link:
+ variables:
+ ROLE: link
+ <<: *lab_dfn
+
+jerry:
+ variables:
+ ROLE: jerry
+ <<: *lab_dfn
+
+minnowmax:
+ variables:
+ ROLE: minnowmax
+ <<: *lab_dfn
+
+opi_pc2:
+ variables:
+ ROLE: opi_pc2
+ <<: *lab_dfn
+
+bpi:
+ variables:
+ ROLE: bpi
+ <<: *lab_dfn
+
+rpi2:
+ variables:
+ ROLE: rpi2
+ <<: *lab_dfn
+
+bob:
+ variables:
+ ROLE: bob
+ <<: *lab_dfn
+
+ff3399:
+ variables:
+ ROLE: ff3399
+ <<: *lab_dfn
+
+coral:
+ variables:
+ ROLE: coral
+ <<: *lab_dfn
+
+rpi3z:
+ variables:
+ ROLE: rpi3z
+ <<: *lab_dfn
+
+bbb:
+ variables:
+ ROLE: bbb
+ <<: *lab_dfn
+
+kevin:
+ variables:
+ ROLE: kevin
+ <<: *lab_dfn
+
+pine64:
+ variables:
+ ROLE: pine64
+ <<: *lab_dfn
+
+c4:
+ variables:
+ ROLE: c4
+ <<: *lab_dfn
+
+rpi4:
+ variables:
+ ROLE: rpi4
+ <<: *lab_dfn
+
+rpi0:
+ variables:
+ ROLE: rpi0
+ <<: *lab_dfn
+
+snow:
+ variables:
+ ROLE: snow
+ <<: *lab_dfn
+
+pcduino3:
+ variables:
+ ROLE: pcduino3
+ <<: *lab_dfn
+
+nyan-big:
+ variables:
+ ROLE: nyan-big
+ <<: *lab_dfn
diff --git a/test/py/conftest.py b/test/py/conftest.py
index 46a410c..d9f074f 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -23,6 +23,7 @@
import pytest
import re
from _pytest.runner import runtestprotocol
+import subprocess
import sys
from u_boot_spawn import BootFail, Timeout, Unexpected, handle_exception
@@ -65,12 +66,16 @@
parser.addoption('--build-dir', default=None,
help='U-Boot build directory (O=)')
+ parser.addoption('--build-dir-extra', default=None,
+ help='U-Boot build directory for extra build (O=)')
parser.addoption('--result-dir', default=None,
help='U-Boot test result/tmp directory')
parser.addoption('--persistent-data-dir', default=None,
help='U-Boot test persistent generated data directory')
parser.addoption('--board-type', '--bd', '-B', default='sandbox',
help='U-Boot board type')
+ parser.addoption('--board-type-extra', '--bde', default='sandbox',
+ help='U-Boot extra board type')
parser.addoption('--board-identity', '--id', default='na',
help='U-Boot board identity/instance')
parser.addoption('--build', default=False, action='store_true',
@@ -80,6 +85,9 @@
parser.addoption('--gdbserver', default=None,
help='Run sandbox under gdbserver. The argument is the channel '+
'over which gdbserver should communicate, e.g. localhost:1234')
+ parser.addoption('--role', help='U-Boot board role (for Labgrid-sjg)')
+ parser.addoption('--use-running-system', default=False, action='store_true',
+ help="Assume that U-Boot is ready and don't wait for a prompt")
def run_build(config, source_dir, build_dir, board_type, log):
"""run_build: Build U-Boot
@@ -125,26 +133,71 @@
Returns:
tuple:
str: Board type (U-Boot build name)
+ str: Extra board type (where two U-Boot builds are needed)
str: Identity for the lab board
str: Build directory
+ str: Extra build directory (where two U-Boot builds are needed)
str: Source directory
"""
- board_type = config.getoption('board_type')
- board_identity = config.getoption('board_identity')
+ role = config.getoption('role')
+
+ # Get a few provided parameters
build_dir = config.getoption('build_dir')
+ build_dir_extra = config.getoption('build_dir_extra')
+ if role:
+ # When using a role, build_dir and build_dir_extra are normally not set,
+ # since they are picked up from Labgrid-sjg via the u-boot-test-getrole
+ # script
+ board_identity = role
+ cmd = ['u-boot-test-getrole', role, '--configure']
+ env = os.environ.copy()
+ if build_dir:
+ env['U_BOOT_BUILD_DIR'] = build_dir
+ if build_dir_extra:
+ env['U_BOOT_BUILD_DIR_EXTRA'] = build_dir_extra
+ proc = subprocess.run(cmd, capture_output=True, encoding='utf-8',
+ env=env)
+ if proc.returncode:
+ raise ValueError(proc.stderr)
+ # For debugging
+ # print('conftest: lab:', proc.stdout)
+ vals = {}
+ for line in proc.stdout.splitlines():
+ item, value = line.split(' ', maxsplit=1)
+ k = item.split(':')[-1]
+ vals[k] = value
+ # For debugging
+ # print('conftest: lab info:', vals)
+
+ # Read the build directories here, in case none were provided in the
+ # command-line arguments
+ (board_type, board_type_extra, default_build_dir,
+ default_build_dir_extra, source_dir) = (vals['board'],
+ vals['board_extra'], vals['build_dir'], vals['build_dir_extra'],
+ vals['source_dir'])
+ else:
+ board_type = config.getoption('board_type')
+ board_type_extra = config.getoption('board_type_extra')
+ board_identity = config.getoption('board_identity')
+
+ source_dir = os.path.dirname(os.path.dirname(TEST_PY_DIR))
+ default_build_dir = source_dir + '/build-' + board_type
+ default_build_dir_extra = source_dir + '/build-' + board_type_extra
- source_dir = os.path.dirname(os.path.dirname(TEST_PY_DIR))
- default_build_dir = source_dir + '/build-' + board_type
+ # Use the provided command-line arguments if present, else fall back to
if not build_dir:
build_dir = default_build_dir
+ if not build_dir_extra:
+ build_dir_extra = default_build_dir_extra
- return board_type, board_identity, build_dir, source_dir
+ return (board_type, board_type_extra, board_identity, build_dir,
+ build_dir_extra, source_dir)
def pytest_xdist_setupnodes(config, specs):
"""Clear out any 'done' file from a previous build"""
global build_done_file
- build_dir = get_details(config)[2]
+ build_dir = get_details(config)[3]
build_done_file = Path(build_dir) / 'build.done'
if build_done_file.exists():
@@ -184,7 +237,8 @@
global console
global ubconfig
- board_type, board_identity, build_dir, source_dir = get_details(config)
+ (board_type, board_type_extra, board_identity, build_dir, build_dir_extra,
+ source_dir) = get_details(config)
board_type_filename = board_type.replace('-', '_')
board_identity_filename = board_identity.replace('-', '_')
@@ -249,20 +303,25 @@
ubconfig.test_py_dir = TEST_PY_DIR
ubconfig.source_dir = source_dir
ubconfig.build_dir = build_dir
+ ubconfig.build_dir_extra = build_dir_extra
ubconfig.result_dir = result_dir
ubconfig.persistent_data_dir = persistent_data_dir
ubconfig.board_type = board_type
+ ubconfig.board_type_extra = board_type_extra
ubconfig.board_identity = board_identity
ubconfig.gdbserver = gdbserver
+ ubconfig.use_running_system = config.getoption('use_running_system')
ubconfig.dtb = build_dir + '/arch/sandbox/dts/test.dtb'
ubconfig.connection_ok = True
env_vars = (
'board_type',
+ 'board_type_extra',
'board_identity',
'source_dir',
'test_py_dir',
'build_dir',
+ 'build_dir_extra',
'result_dir',
'persistent_data_dir',
)
diff --git a/test/py/tests/test_spi.py b/test/py/tests/test_spi.py
index caca930..44e5473 100644
--- a/test/py/tests/test_spi.py
+++ b/test/py/tests/test_spi.py
@@ -695,7 +695,7 @@
# Read to relocation address
output = u_boot_console.run_command('bdinfo')
- m = re.search('relocaddr\s*= (.+)', output)
+ m = re.search(r'relocaddr\s*= (.+)', output)
res_area = int(m.group(1), 16)
start = 0
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index d8d0bdf..fa9cd57 100644
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -23,12 +23,22 @@
pattern_unknown_command = re.compile('Unknown command \'.*\' - try \'help\'')
pattern_error_notification = re.compile('## Error: ')
pattern_error_please_reset = re.compile('### ERROR ### Please RESET the board ###')
+pattern_ready_prompt = re.compile('{lab ready in (.*)s: (.*)}')
+pattern_lab_mode = re.compile('{lab mode.*}')
PAT_ID = 0
PAT_RE = 1
# Timeout before expecting the console to be ready (in milliseconds)
-TIMEOUT_MS = 30000
+TIMEOUT_MS = 30000 # Standard timeout
+TIMEOUT_CMD_MS = 10000 # Command-echo timeout
+
+# Timeout for board preparation in lab mode. This needs to be enough to build
+# U-Boot, write it to the board and then boot the board. Since this process is
+# under the control of another program (e.g. Labgrid), it will failure sooner
+# if something goes way. So use a very long timeout here to cover all possible
+# situations.
+TIMEOUT_PREPARE_MS = 3 * 60 * 1000
bad_pattern_defs = (
('spl_signon', pattern_u_boot_spl_signon),
@@ -142,6 +152,7 @@
self.at_prompt = False
self.at_prompt_logevt = None
+ self.lab_mode = False
def get_spawn(self):
# This is not called, ssubclass must define this.
@@ -172,43 +183,75 @@
"""
if self.p:
- self.p.close()
+ self.log.start_section('Stopping U-Boot')
+ close_type = self.p.close()
+ self.log.info(f'Close type: {close_type}')
+ self.log.end_section('Stopping U-Boot')
self.logstream.close()
+ def set_lab_mode(self):
+ """Select lab mode
+
+ This tells us that we will get a 'lab ready' message when the board is
+ ready for use. We don't need to look for signon messages.
+ """
+ self.log.info(f'test.py: Lab mode is active')
+ self.p.timeout = TIMEOUT_PREPARE_MS
+ self.lab_mode = True
+
def wait_for_boot_prompt(self, loop_num = 1):
"""Wait for the boot up until command prompt. This is for internal use only.
"""
try:
+ self.log.info('Waiting for U-Boot to be ready')
bcfg = self.config.buildconfig
config_spl_serial = bcfg.get('config_spl_serial', 'n') == 'y'
env_spl_skipped = self.config.env.get('env__spl_skipped', False)
env_spl_banner_times = self.config.env.get('env__spl_banner_times', 1)
- while loop_num > 0:
+ while not self.lab_mode and loop_num > 0:
loop_num -= 1
while config_spl_serial and not env_spl_skipped and env_spl_banner_times > 0:
- m = self.p.expect([pattern_u_boot_spl_signon] +
- self.bad_patterns)
- if m != 0:
+ m = self.p.expect([pattern_u_boot_spl_signon,
+ pattern_lab_mode] + self.bad_patterns)
+ if m == 1:
+ self.set_lab_mode()
+ break
+ elif m != 0:
raise BootFail('Bad pattern found on SPL console: ' +
- self.bad_pattern_ids[m - 1])
+ self.bad_pattern_ids[m - 1])
env_spl_banner_times -= 1
- m = self.p.expect([pattern_u_boot_main_signon] + self.bad_patterns)
- if m != 0:
- raise BootFail('Bad pattern found on console: ' +
- self.bad_pattern_ids[m - 1])
- self.u_boot_version_string = self.p.after
+ if not self.lab_mode:
+ m = self.p.expect([pattern_u_boot_main_signon,
+ pattern_lab_mode] + self.bad_patterns)
+ if m == 1:
+ self.set_lab_mode()
+ elif m != 0:
+ raise BootFail('Bad pattern found on console: ' +
+ self.bad_pattern_ids[m - 1])
+ if not self.lab_mode:
+ self.u_boot_version_string = self.p.after
while True:
- m = self.p.expect([self.prompt_compiled,
+ m = self.p.expect([self.prompt_compiled, pattern_ready_prompt,
pattern_stop_autoboot_prompt] + self.bad_patterns)
if m == 0:
+ self.log.info(f'Found ready prompt {m}')
break
- if m == 1:
+ elif m == 1:
+ m = pattern_ready_prompt.search(self.p.after)
+ self.u_boot_version_string = m.group(2)
+ self.log.info(f'Lab: Board is ready')
+ self.p.timeout = TIMEOUT_MS
+ break
+ if m == 2:
+ self.log.info(f'Found autoboot prompt {m}')
self.p.send(' ')
continue
- raise BootFail('Bad pattern found on console: ' +
- self.bad_pattern_ids[m - 2])
+ if not self.lab_mode:
+ raise BootFail('Missing prompt / ready message on console: ' +
+ self.bad_pattern_ids[m - 3])
+ self.log.info(f'U-Boot is ready')
finally:
self.log.timestamp()
@@ -261,22 +304,28 @@
try:
self.at_prompt = False
+ if not self.p:
+ raise BootFail(
+ f"Lab failure: Connection lost when sending command '{cmd}'")
+
if send_nl:
cmd += '\n'
- while cmd:
- # Limit max outstanding data, so UART FIFOs don't overflow
- chunk = cmd[:self.max_fifo_fill]
- cmd = cmd[self.max_fifo_fill:]
- self.p.send(chunk)
- if not wait_for_echo:
- continue
- chunk = re.escape(chunk)
- chunk = chunk.replace('\\\n', '[\r\n]')
- m = self.p.expect([chunk] + self.bad_patterns)
- if m != 0:
- self.at_prompt = False
- raise BootFail('Bad pattern found on console: ' +
- self.bad_pattern_ids[m - 1])
+ rem = cmd # Remaining to be sent
+ with self.temporary_timeout(TIMEOUT_CMD_MS):
+ while rem:
+ # Limit max outstanding data, so UART FIFOs don't overflow
+ chunk = rem[:self.max_fifo_fill]
+ rem = rem[self.max_fifo_fill:]
+ self.p.send(chunk)
+ if not wait_for_echo:
+ continue
+ chunk = re.escape(chunk)
+ chunk = chunk.replace('\\\n', '[\r\n]')
+ m = self.p.expect([chunk] + self.bad_patterns)
+ if m != 0:
+ self.at_prompt = False
+ raise BootFail(f"Failed to get echo on console (cmd '{cmd}':rem '{rem}'): " +
+ self.bad_pattern_ids[m - 1])
if not wait_for_prompt:
return
if wait_for_reboot:
@@ -440,11 +489,17 @@
if not self.config.gdbserver:
self.p.timeout = TIMEOUT_MS
self.p.logfile_read = self.logstream
- if expect_reset:
- loop_num = 2
+ if self.config.use_running_system:
+ # Send an empty command to set up the 'expect' logic. This has
+ # the side effect of ensuring that there was no partial command
+ # line entered
+ self.run_command(' ')
else:
- loop_num = 1
- self.wait_for_boot_prompt(loop_num = loop_num)
+ if expect_reset:
+ loop_num = 2
+ else:
+ loop_num = 1
+ self.wait_for_boot_prompt(loop_num = loop_num)
self.at_prompt = True
self.at_prompt_logevt = self.logstream.logfile.cur_evt
except Exception as ex:
diff --git a/test/py/u_boot_console_exec_attach.py b/test/py/u_boot_console_exec_attach.py
index 8dd8cc1..8b253b4 100644
--- a/test/py/u_boot_console_exec_attach.py
+++ b/test/py/u_boot_console_exec_attach.py
@@ -59,14 +59,27 @@
args = [self.config.board_type, self.config.board_identity]
s = Spawn(['u-boot-test-console'] + args)
- try:
- self.log.action('Resetting board')
- cmd = ['u-boot-test-reset'] + args
- runner = self.log.get_runner(cmd[0], sys.stdout)
- runner.run(cmd)
- runner.close()
- except:
- s.close()
- raise
+ if self.config.use_running_system:
+ self.log.action('Connecting to board without reset')
+ else:
+ try:
+ self.log.action('Resetting board')
+ cmd = ['u-boot-test-reset'] + args
+ runner = self.log.get_runner(cmd[0], sys.stdout)
+ runner.run(cmd)
+ runner.close()
+ except:
+ s.close()
+ raise
return s
+
+ def close(self):
+ super().close()
+
+ self.log.action('Releasing board')
+ args = [self.config.board_type, self.config.board_identity]
+ cmd = ['u-boot-test-release'] + args
+ runner = self.log.get_runner(cmd[0], sys.stdout)
+ runner.run(cmd)
+ runner.close()
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 24d3690..c703454 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -5,15 +5,21 @@
Logic to spawn a sub-process and interact with its stdio.
"""
+import io
import os
import re
import pty
import pytest
import signal
import select
+import sys
+import termios
import time
import traceback
+# Character to send (twice) to exit the terminal
+EXIT_CHAR = 0x1d # FS (Ctrl + ])
+
class Timeout(Exception):
"""An exception sub-class that indicates that a timeout occurred."""
@@ -115,11 +121,30 @@
finally:
os._exit(255)
+ old = None
try:
+ isatty = False
+ try:
+ isatty = os.isatty(sys.stdout.fileno())
+
+ # with --capture=tee-sys we cannot call fileno()
+ except io.UnsupportedOperation as exc:
+ pass
+ if isatty:
+ new = termios.tcgetattr(self.fd)
+ old = new
+ new[3] = new[3] & ~(termios.ICANON | termios.ISIG)
+ new[3] = new[3] & ~termios.ECHO
+ new[6][termios.VMIN] = 0
+ new[6][termios.VTIME] = 0
+ termios.tcsetattr(self.fd, termios.TCSANOW, new)
+
self.poll = select.poll()
self.poll.register(self.fd, select.POLLIN | select.POLLPRI | select.POLLERR |
select.POLLHUP | select.POLLNVAL)
except:
+ if old:
+ termios.tcsetattr(self.fd, termios.TCSANOW, old)
self.close()
raise
@@ -289,15 +314,28 @@
None.
Returns:
- Nothing.
+ str: Type of closure completed
"""
+ # For Labgrid-sjg, ask it is exit gracefully, so it can transition the
+ # board to the final state (like 'off') before exiting.
+ if os.environ.get('USE_LABGRID_SJG'):
+ self.send(chr(EXIT_CHAR) * 2)
+ # Wait about 10 seconds for Labgrid to close and power off the board
+ for _ in range(100):
+ if not self.isalive():
+ return 'normal'
+ time.sleep(0.1)
+
+ # That didn't work, so try closing the PTY
os.close(self.fd)
for _ in range(100):
if not self.isalive():
- break
+ return 'break'
time.sleep(0.1)
+ return 'timeout'
+
def get_expect_output(self):
"""Return the output read by expect()