buildman: Use bytes for the environment
At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.
Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.
Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index acb5a29..fd137f7 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -179,27 +179,35 @@
output and possibly unicode encoded output of all build tools by
adding LC_ALL=C.
+ Note that os.environb is used to obtain the environment, since in some
+ cases the environment many contain non-ASCII characters and we see
+ errors like:
+
+ UnicodeEncodeError: 'utf-8' codec can't encode characters in position
+ 569-570: surrogates not allowed
+
Args:
full_path: Return the full path in CROSS_COMPILE and don't set
PATH
Returns:
- Dict containing the environemnt to use. This is based on the current
- environment, with changes as needed to CROSS_COMPILE, PATH and
- LC_ALL.
+ Dict containing the (bytes) environment to use. This is based on the
+ current environment, with changes as needed to CROSS_COMPILE, PATH
+ and LC_ALL.
"""
- env = dict(os.environ)
+ env = dict(os.environb)
wrapper = self.GetWrapper()
if self.override_toolchain:
# We'll use MakeArgs() to provide this
pass
elif full_path:
- env['CROSS_COMPILE'] = wrapper + os.path.join(self.path, self.cross)
+ env[b'CROSS_COMPILE'] = tools.ToBytes(
+ wrapper + os.path.join(self.path, self.cross))
else:
- env['CROSS_COMPILE'] = wrapper + self.cross
- env['PATH'] = self.path + ':' + env['PATH']
+ env[b'CROSS_COMPILE'] = tools.ToBytes(wrapper + self.cross)
+ env[b'PATH'] = tools.ToBytes(self.path) + b':' + env[b'PATH']
- env['LC_ALL'] = 'C'
+ env[b'LC_ALL'] = b'C'
return env