blob: 07136f56f57fd332dab417d0b158de38caa87216 [file] [log] [blame]
Masahiro Yamadab6160812015-05-20 11:36:07 +09001#!/usr/bin/env python2
2#
3# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
4#
5# SPDX-License-Identifier: GPL-2.0+
6#
7
8"""
9Move config options from headers to defconfig files.
10
11Since Kconfig was introduced to U-Boot, we have worked on moving
12config options from headers to Kconfig (defconfig).
13
14This tool intends to help this tremendous work.
15
16
17Usage
18-----
19
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090020First, you must edit the Kconfig to add the menu entries for the configs
Joe Hershberger166edec2015-05-19 13:21:17 -050021you are moving.
22
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090023And then run this tool giving CONFIG names you want to move.
24For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25simply type as follows:
Masahiro Yamadab6160812015-05-20 11:36:07 +090026
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090027 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
Masahiro Yamadab6160812015-05-20 11:36:07 +090028
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090029The tool walks through all the defconfig files and move the given CONFIGs.
Masahiro Yamadab6160812015-05-20 11:36:07 +090030
31The log is also displayed on the terminal.
32
33Each line is printed in the format
34<defconfig_name> : <action>
35
36<defconfig_name> is the name of the defconfig
37(without the suffix _defconfig).
38
39<action> shows what the tool did for that defconfig.
40It looks like one of the followings:
41
42 - Move 'CONFIG_... '
43 This config option was moved to the defconfig
44
Masahiro Yamada5643d6e2016-05-19 15:51:56 +090045 - CONFIG_... is not defined in Kconfig. Do nothing.
46 The entry for this CONFIG was not found in Kconfig.
47 There are two common cases:
48 - You forgot to create an entry for the CONFIG before running
49 this tool, or made a typo in a CONFIG passed to this tool.
50 - The entry was hidden due to unmet 'depends on'.
51 This is correct behavior.
Masahiro Yamadab6160812015-05-20 11:36:07 +090052
Masahiro Yamada5643d6e2016-05-19 15:51:56 +090053 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
54 The define in the config header matched the one in Kconfig.
55 We do not need to touch it.
Masahiro Yamadab6160812015-05-20 11:36:07 +090056
57 - Undefined. Do nothing.
58 This config option was not found in the config header.
59 Nothing to do.
60
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +090061 - Compiler is missing. Do nothing.
62 The compiler specified for this architecture was not found
63 in your PATH environment.
64 (If -e option is passed, the tool exits immediately.)
65
66 - Failed to process.
Masahiro Yamadab6160812015-05-20 11:36:07 +090067 An error occurred during processing this defconfig. Skipped.
68 (If -e option is passed, the tool exits immediately on error.)
69
70Finally, you will be asked, Clean up headers? [y/n]:
71
72If you say 'y' here, the unnecessary config defines are removed
73from the config headers (include/configs/*.h).
74It just uses the regex method, so you should not rely on it.
75Just in case, please do 'git diff' to see what happened.
76
77
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090078How does it work?
79-----------------
Masahiro Yamadab6160812015-05-20 11:36:07 +090080
81This tool runs configuration and builds include/autoconf.mk for every
82defconfig. The config options defined in Kconfig appear in the .config
83file (unless they are hidden because of unmet dependency.)
84On the other hand, the config options defined by board headers are seen
85in include/autoconf.mk. The tool looks for the specified options in both
Masahiro Yamadab903c4e2016-05-19 15:51:58 +090086of them to decide the appropriate action for the options. If the given
87config option is found in the .config, but its value does not match the
88one from the board header, the config option in the .config is replaced
89with the define in the board header. Then, the .config is synced by
90"make savedefconfig" and the defconfig is updated with it.
Masahiro Yamadab6160812015-05-20 11:36:07 +090091
92For faster processing, this tool handles multi-threading. It creates
93separate build directories where the out-of-tree build is run. The
94temporary build directories are automatically created and deleted as
95needed. The number of threads are chosen based on the number of the CPU
96cores of your system although you can change it via -j (--jobs) option.
97
98
99Toolchains
100----------
101
102Appropriate toolchain are necessary to generate include/autoconf.mk
103for all the architectures supported by U-Boot. Most of them are available
104at the kernel.org site, some are not provided by kernel.org.
105
106The default per-arch CROSS_COMPILE used by this tool is specified by
107the list below, CROSS_COMPILE. You may wish to update the list to
108use your own. Instead of modifying the list directly, you can give
109them via environments.
110
111
112Available options
113-----------------
114
115 -c, --color
116 Surround each portion of the log with escape sequences to display it
117 in color on the terminal.
118
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500119 -d, --defconfigs
120 Specify a file containing a list of defconfigs to move
121
Masahiro Yamadab6160812015-05-20 11:36:07 +0900122 -n, --dry-run
Masahiro Yamadab903c4e2016-05-19 15:51:58 +0900123 Perform a trial run that does not make any changes. It is useful to
Masahiro Yamadab6160812015-05-20 11:36:07 +0900124 see what is going to happen before one actually runs it.
125
126 -e, --exit-on-error
127 Exit immediately if Make exits with a non-zero status while processing
128 a defconfig file.
129
Joe Hershberger23475932015-05-19 13:21:20 -0500130 -H, --headers-only
131 Only cleanup the headers; skip the defconfig processing
132
Masahiro Yamadab6160812015-05-20 11:36:07 +0900133 -j, --jobs
134 Specify the number of threads to run simultaneously. If not specified,
135 the number of threads is the same as the number of CPU cores.
136
Joe Hershberger808b63f2015-05-19 13:21:24 -0500137 -v, --verbose
138 Show any build errors as boards are built
139
Masahiro Yamadab6160812015-05-20 11:36:07 +0900140To see the complete list of supported options, run
141
142 $ tools/moveconfig.py -h
143
144"""
145
146import fnmatch
147import multiprocessing
148import optparse
149import os
150import re
151import shutil
152import subprocess
153import sys
154import tempfile
155import time
156
157SHOW_GNU_MAKE = 'scripts/show-gnu-make'
158SLEEP_TIME=0.03
159
160# Here is the list of cross-tools I use.
161# Most of them are available at kernel.org
162# (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the followings:
163# arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
164# blackfin: http://sourceforge.net/projects/adi-toolchain/files/
Bin Meng3bb02f62015-09-25 01:22:39 -0700165# nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
Masahiro Yamadab6160812015-05-20 11:36:07 +0900166# nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
167# sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
Bin Meng07a50f92016-02-21 21:18:02 -0800168#
169# openrisc kernel.org toolchain is out of date, download latest one from
170# http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions
Masahiro Yamadab6160812015-05-20 11:36:07 +0900171CROSS_COMPILE = {
172 'arc': 'arc-linux-',
173 'aarch64': 'aarch64-linux-',
174 'arm': 'arm-unknown-linux-gnueabi-',
175 'avr32': 'avr32-linux-',
176 'blackfin': 'bfin-elf-',
177 'm68k': 'm68k-linux-',
178 'microblaze': 'microblaze-linux-',
179 'mips': 'mips-linux-',
180 'nds32': 'nds32le-linux-',
181 'nios2': 'nios2-linux-gnu-',
Bin Meng07a50f92016-02-21 21:18:02 -0800182 'openrisc': 'or1k-elf-',
Masahiro Yamadab6160812015-05-20 11:36:07 +0900183 'powerpc': 'powerpc-linux-',
184 'sh': 'sh-linux-gnu-',
185 'sparc': 'sparc-linux-',
186 'x86': 'i386-linux-'
187}
188
189STATE_IDLE = 0
190STATE_DEFCONFIG = 1
191STATE_AUTOCONF = 2
Joe Hershberger166edec2015-05-19 13:21:17 -0500192STATE_SAVEDEFCONFIG = 3
Masahiro Yamadab6160812015-05-20 11:36:07 +0900193
194ACTION_MOVE = 0
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900195ACTION_NO_ENTRY = 1
196ACTION_NO_CHANGE = 2
Masahiro Yamadab6160812015-05-20 11:36:07 +0900197
198COLOR_BLACK = '0;30'
199COLOR_RED = '0;31'
200COLOR_GREEN = '0;32'
201COLOR_BROWN = '0;33'
202COLOR_BLUE = '0;34'
203COLOR_PURPLE = '0;35'
204COLOR_CYAN = '0;36'
205COLOR_LIGHT_GRAY = '0;37'
206COLOR_DARK_GRAY = '1;30'
207COLOR_LIGHT_RED = '1;31'
208COLOR_LIGHT_GREEN = '1;32'
209COLOR_YELLOW = '1;33'
210COLOR_LIGHT_BLUE = '1;34'
211COLOR_LIGHT_PURPLE = '1;35'
212COLOR_LIGHT_CYAN = '1;36'
213COLOR_WHITE = '1;37'
214
215### helper functions ###
216def get_devnull():
217 """Get the file object of '/dev/null' device."""
218 try:
219 devnull = subprocess.DEVNULL # py3k
220 except AttributeError:
221 devnull = open(os.devnull, 'wb')
222 return devnull
223
224def check_top_directory():
225 """Exit if we are not at the top of source directory."""
226 for f in ('README', 'Licenses'):
227 if not os.path.exists(f):
228 sys.exit('Please run at the top of source directory.')
229
Masahiro Yamada990e6772016-05-19 15:51:54 +0900230def check_clean_directory():
231 """Exit if the source tree is not clean."""
232 for f in ('.config', 'include/config'):
233 if os.path.exists(f):
234 sys.exit("source tree is not clean, please run 'make mrproper'")
235
Masahiro Yamadab6160812015-05-20 11:36:07 +0900236def get_make_cmd():
237 """Get the command name of GNU Make.
238
239 U-Boot needs GNU Make for building, but the command name is not
240 necessarily "make". (for example, "gmake" on FreeBSD).
241 Returns the most appropriate command name on your system.
242 """
243 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
244 ret = process.communicate()
245 if process.returncode:
246 sys.exit('GNU Make not found')
247 return ret[0].rstrip()
248
249def color_text(color_enabled, color, string):
250 """Return colored string."""
251 if color_enabled:
252 return '\033[' + color + 'm' + string + '\033[0m'
253 else:
254 return string
255
256def log_msg(color_enabled, color, defconfig, msg):
257 """Return the formated line for the log."""
258 return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \
259 color_text(color_enabled, color, msg) + '\n'
260
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900261def update_cross_compile(color_enabled):
Robert P. J. Day8c60f922016-05-04 04:47:31 -0400262 """Update per-arch CROSS_COMPILE via environment variables
Masahiro Yamadab6160812015-05-20 11:36:07 +0900263
264 The default CROSS_COMPILE values are available
265 in the CROSS_COMPILE list above.
266
Robert P. J. Day8c60f922016-05-04 04:47:31 -0400267 You can override them via environment variables
Masahiro Yamadab6160812015-05-20 11:36:07 +0900268 CROSS_COMPILE_{ARCH}.
269
270 For example, if you want to override toolchain prefixes
271 for ARM and PowerPC, you can do as follows in your shell:
272
273 export CROSS_COMPILE_ARM=...
274 export CROSS_COMPILE_POWERPC=...
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900275
276 Then, this function checks if specified compilers really exist in your
277 PATH environment.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900278 """
279 archs = []
280
281 for arch in os.listdir('arch'):
282 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
283 archs.append(arch)
284
285 # arm64 is a special case
286 archs.append('aarch64')
287
288 for arch in archs:
289 env = 'CROSS_COMPILE_' + arch.upper()
290 cross_compile = os.environ.get(env)
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900291 if not cross_compile:
292 cross_compile = CROSS_COMPILE.get(arch, '')
293
294 for path in os.environ["PATH"].split(os.pathsep):
295 gcc_path = os.path.join(path, cross_compile + 'gcc')
296 if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
297 break
298 else:
299 print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
300 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
301 % (cross_compile, arch))
302 cross_compile = None
303
304 CROSS_COMPILE[arch] = cross_compile
Masahiro Yamadab6160812015-05-20 11:36:07 +0900305
306def cleanup_one_header(header_path, patterns, dry_run):
307 """Clean regex-matched lines away from a file.
308
309 Arguments:
310 header_path: path to the cleaned file.
311 patterns: list of regex patterns. Any lines matching to these
312 patterns are deleted.
313 dry_run: make no changes, but still display log.
314 """
315 with open(header_path) as f:
316 lines = f.readlines()
317
318 matched = []
319 for i, line in enumerate(lines):
320 for pattern in patterns:
321 m = pattern.search(line)
322 if m:
323 print '%s: %s: %s' % (header_path, i + 1, line),
324 matched.append(i)
325 break
326
327 if dry_run or not matched:
328 return
329
330 with open(header_path, 'w') as f:
331 for i, line in enumerate(lines):
332 if not i in matched:
333 f.write(line)
334
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900335def cleanup_headers(configs, dry_run):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900336 """Delete config defines from board headers.
337
338 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900339 configs: A list of CONFIGs to remove.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900340 dry_run: make no changes, but still display log.
341 """
342 while True:
343 choice = raw_input('Clean up headers? [y/n]: ').lower()
344 print choice
345 if choice == 'y' or choice == 'n':
346 break
347
348 if choice == 'n':
349 return
350
351 patterns = []
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900352 for config in configs:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900353 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
354 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
355
Joe Hershbergerb78ad422015-05-19 13:21:21 -0500356 for dir in 'include', 'arch', 'board':
357 for (dirpath, dirnames, filenames) in os.walk(dir):
358 for filename in filenames:
359 if not fnmatch.fnmatch(filename, '*~'):
360 cleanup_one_header(os.path.join(dirpath, filename),
361 patterns, dry_run)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900362
363### classes ###
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900364class Progress:
365
366 """Progress Indicator"""
367
368 def __init__(self, total):
369 """Create a new progress indicator.
370
371 Arguments:
372 total: A number of defconfig files to process.
373 """
374 self.current = 0
375 self.total = total
376
377 def inc(self):
378 """Increment the number of processed defconfig files."""
379
380 self.current += 1
381
382 def show(self):
383 """Display the progress."""
384 print ' %d defconfigs out of %d\r' % (self.current, self.total),
385 sys.stdout.flush()
386
Masahiro Yamadab6160812015-05-20 11:36:07 +0900387class KconfigParser:
388
389 """A parser of .config and include/autoconf.mk."""
390
391 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
392 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
393
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900394 def __init__(self, configs, options, progress, build_dir):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900395 """Create a new parser.
396
397 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900398 configs: A list of CONFIGs to move.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900399 options: option flags.
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900400 progress: A progress indicator
Masahiro Yamadab6160812015-05-20 11:36:07 +0900401 build_dir: Build directory.
402 """
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900403 self.configs = configs
Masahiro Yamadab6160812015-05-20 11:36:07 +0900404 self.options = options
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900405 self.progress = progress
Masahiro Yamadab6160812015-05-20 11:36:07 +0900406 self.build_dir = build_dir
407
408 def get_cross_compile(self):
409 """Parse .config file and return CROSS_COMPILE.
410
411 Returns:
412 A string storing the compiler prefix for the architecture.
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900413 Return a NULL string for architectures that do not require
414 compiler prefix (Sandbox and native build is the case).
415 Return None if the specified compiler is missing in your PATH.
416 Caller should distinguish '' and None.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900417 """
418 arch = ''
419 cpu = ''
420 dotconfig = os.path.join(self.build_dir, '.config')
421 for line in open(dotconfig):
422 m = self.re_arch.match(line)
423 if m:
424 arch = m.group(1)
425 continue
426 m = self.re_cpu.match(line)
427 if m:
428 cpu = m.group(1)
429
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900430 if not arch:
431 return None
Masahiro Yamadab6160812015-05-20 11:36:07 +0900432
433 # fix-up for aarch64
434 if arch == 'arm' and cpu == 'armv8':
435 arch = 'aarch64'
436
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900437 return CROSS_COMPILE.get(arch, None)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900438
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900439 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900440 """Parse .config, defconfig, include/autoconf.mk for one config.
441
442 This function looks for the config options in the lines from
443 defconfig, .config, and include/autoconf.mk in order to decide
444 which action should be taken for this defconfig.
445
446 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900447 config: CONFIG name to parse.
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900448 dotconfig_lines: lines from the .config file.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900449 autoconf_lines: lines from the include/autoconf.mk file.
450
451 Returns:
452 A tupple of the action for this defconfig and the line
453 matched for the config.
454 """
Masahiro Yamadab6160812015-05-20 11:36:07 +0900455 not_set = '# %s is not set' % config
456
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900457 for line in dotconfig_lines:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900458 line = line.rstrip()
459 if line.startswith(config + '=') or line == not_set:
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900460 old_val = line
461 break
Masahiro Yamadab6160812015-05-20 11:36:07 +0900462 else:
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900463 return (ACTION_NO_ENTRY, config)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900464
465 for line in autoconf_lines:
466 line = line.rstrip()
467 if line.startswith(config + '='):
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900468 new_val = line
Masahiro Yamadab6160812015-05-20 11:36:07 +0900469 break
Masahiro Yamadab6160812015-05-20 11:36:07 +0900470 else:
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900471 new_val = not_set
472
473 if old_val == new_val:
474 return (ACTION_NO_CHANGE, new_val)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900475
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900476 # If this CONFIG is neither bool nor trisate
477 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
478 # tools/scripts/define2mk.sed changes '1' to 'y'.
479 # This is a problem if the CONFIG is int type.
480 # Check the type in Kconfig and handle it correctly.
481 if new_val[-2:] == '=y':
482 new_val = new_val[:-1] + '1'
483
484 return (ACTION_MOVE, new_val)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900485
Masahiro Yamada7c0d9d22016-05-19 15:51:50 +0900486 def update_dotconfig(self, defconfig):
487 """Parse files for the config options and update the .config.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900488
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900489 This function parses the generated .config and include/autoconf.mk
490 searching the target options.
Masahiro Yamada7c0d9d22016-05-19 15:51:50 +0900491 Move the config option(s) to the .config as needed.
492 Also, display the log to show what happened to the .config.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900493
494 Arguments:
495 defconfig: defconfig name.
496 """
497
Masahiro Yamadab6160812015-05-20 11:36:07 +0900498 dotconfig_path = os.path.join(self.build_dir, '.config')
499 autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk')
500 results = []
501
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900502 with open(dotconfig_path) as f:
503 dotconfig_lines = f.readlines()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900504
Masahiro Yamadab6160812015-05-20 11:36:07 +0900505 with open(autoconf_path) as f:
506 autoconf_lines = f.readlines()
507
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900508 for config in self.configs:
509 result = self.parse_one_config(config, dotconfig_lines,
Joe Hershberger166edec2015-05-19 13:21:17 -0500510 autoconf_lines)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900511 results.append(result)
512
513 log = ''
514
515 for (action, value) in results:
516 if action == ACTION_MOVE:
517 actlog = "Move '%s'" % value
518 log_color = COLOR_LIGHT_GREEN
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900519 elif action == ACTION_NO_ENTRY:
520 actlog = "%s is not defined in Kconfig. Do nothing." % value
Masahiro Yamadab6160812015-05-20 11:36:07 +0900521 log_color = COLOR_LIGHT_BLUE
Masahiro Yamada5643d6e2016-05-19 15:51:56 +0900522 elif action == ACTION_NO_CHANGE:
523 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
524 % value
Masahiro Yamadab6160812015-05-20 11:36:07 +0900525 log_color = COLOR_LIGHT_PURPLE
Masahiro Yamadab6160812015-05-20 11:36:07 +0900526 else:
527 sys.exit("Internal Error. This should not happen.")
528
529 log += log_msg(self.options.color, log_color, defconfig, actlog)
530
531 # Some threads are running in parallel.
532 # Print log in one shot to not mix up logs from different threads.
533 print log,
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900534 self.progress.show()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900535
Masahiro Yamada953d93b2016-05-19 15:51:49 +0900536 with open(dotconfig_path, 'a') as f:
537 for (action, value) in results:
538 if action == ACTION_MOVE:
539 f.write(value + '\n')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900540
541 os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf'))
542 os.remove(autoconf_path)
543
544class Slot:
545
546 """A slot to store a subprocess.
547
548 Each instance of this class handles one subprocess.
549 This class is useful to control multiple threads
550 for faster processing.
551 """
552
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900553 def __init__(self, configs, options, progress, devnull, make_cmd):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900554 """Create a new process slot.
555
556 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900557 configs: A list of CONFIGs to move.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900558 options: option flags.
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900559 progress: A progress indicator.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900560 devnull: A file object of '/dev/null'.
561 make_cmd: command name of GNU Make.
562 """
563 self.options = options
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900564 self.progress = progress
Masahiro Yamadab6160812015-05-20 11:36:07 +0900565 self.build_dir = tempfile.mkdtemp()
566 self.devnull = devnull
567 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900568 self.parser = KconfigParser(configs, options, progress, self.build_dir)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900569 self.state = STATE_IDLE
570 self.failed_boards = []
571
572 def __del__(self):
573 """Delete the working directory
574
575 This function makes sure the temporary directory is cleaned away
576 even if Python suddenly dies due to error. It should be done in here
577 because it is guranteed the destructor is always invoked when the
578 instance of the class gets unreferenced.
579
580 If the subprocess is still running, wait until it finishes.
581 """
582 if self.state != STATE_IDLE:
583 while self.ps.poll() == None:
584 pass
585 shutil.rmtree(self.build_dir)
586
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900587 def add(self, defconfig):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900588 """Assign a new subprocess for defconfig and add it to the slot.
589
590 If the slot is vacant, create a new subprocess for processing the
591 given defconfig and add it to the slot. Just returns False if
592 the slot is occupied (i.e. the current subprocess is still running).
593
594 Arguments:
595 defconfig: defconfig name.
596
597 Returns:
598 Return True on success or False on failure
599 """
600 if self.state != STATE_IDLE:
601 return False
602 cmd = list(self.make_cmd)
603 cmd.append(defconfig)
Joe Hershberger11b02702015-05-19 13:21:23 -0500604 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
605 stderr=subprocess.PIPE)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900606 self.defconfig = defconfig
607 self.state = STATE_DEFCONFIG
608 return True
609
610 def poll(self):
611 """Check the status of the subprocess and handle it as needed.
612
613 Returns True if the slot is vacant (i.e. in idle state).
614 If the configuration is successfully finished, assign a new
615 subprocess to build include/autoconf.mk.
616 If include/autoconf.mk is generated, invoke the parser to
617 parse the .config and the include/autoconf.mk, and then set the
618 slot back to the idle state.
619
620 Returns:
621 Return True if the subprocess is terminated, False otherwise
622 """
623 if self.state == STATE_IDLE:
624 return True
625
626 if self.ps.poll() == None:
627 return False
628
629 if self.ps.poll() != 0:
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900630 print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED,
631 self.defconfig, "Failed to process."),
Joe Hershberger808b63f2015-05-19 13:21:24 -0500632 if self.options.verbose:
633 print >> sys.stderr, color_text(self.options.color,
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900634 COLOR_LIGHT_CYAN,
635 self.ps.stderr.read())
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900636 self.progress.inc()
637 self.progress.show()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900638 if self.options.exit_on_error:
639 sys.exit("Exit on error.")
Masahiro Yamada6ef8ad92016-05-19 15:51:51 +0900640 # If --exit-on-error flag is not set, skip this board and continue.
641 # Record the failed board.
642 self.failed_boards.append(self.defconfig)
643 self.state = STATE_IDLE
644 return True
Masahiro Yamadab6160812015-05-20 11:36:07 +0900645
646 if self.state == STATE_AUTOCONF:
Masahiro Yamada7c0d9d22016-05-19 15:51:50 +0900647 self.parser.update_dotconfig(self.defconfig)
Joe Hershberger166edec2015-05-19 13:21:17 -0500648
649 """Save off the defconfig in a consistent way"""
650 cmd = list(self.make_cmd)
651 cmd.append('savedefconfig')
652 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
Joe Hershberger11b02702015-05-19 13:21:23 -0500653 stderr=subprocess.PIPE)
Joe Hershberger166edec2015-05-19 13:21:17 -0500654 self.state = STATE_SAVEDEFCONFIG
655 return False
656
657 if self.state == STATE_SAVEDEFCONFIG:
Masahiro Yamada953d93b2016-05-19 15:51:49 +0900658 if not self.options.dry_run:
659 shutil.move(os.path.join(self.build_dir, 'defconfig'),
660 os.path.join('configs', self.defconfig))
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900661 self.progress.inc()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900662 self.state = STATE_IDLE
663 return True
664
Joe Hershberger11b02702015-05-19 13:21:23 -0500665 self.cross_compile = self.parser.get_cross_compile()
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900666 if self.cross_compile is None:
667 print >> sys.stderr, log_msg(self.options.color, COLOR_YELLOW,
668 self.defconfig,
669 "Compiler is missing. Do nothing."),
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900670 self.progress.inc()
671 self.progress.show()
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900672 if self.options.exit_on_error:
673 sys.exit("Exit on error.")
674 # If --exit-on-error flag is not set, skip this board and continue.
675 # Record the failed board.
676 self.failed_boards.append(self.defconfig)
677 self.state = STATE_IDLE
678 return True
679
Masahiro Yamadab6160812015-05-20 11:36:07 +0900680 cmd = list(self.make_cmd)
Joe Hershberger11b02702015-05-19 13:21:23 -0500681 if self.cross_compile:
682 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
Joe Hershberger765442b2015-05-19 13:21:18 -0500683 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900684 cmd.append('include/config/auto.conf')
Joe Hershberger11b02702015-05-19 13:21:23 -0500685 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
Joe Hershberger11b02702015-05-19 13:21:23 -0500686 stderr=subprocess.PIPE)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900687 self.state = STATE_AUTOCONF
688 return False
689
690 def get_failed_boards(self):
691 """Returns a list of failed boards (defconfigs) in this slot.
692 """
693 return self.failed_boards
694
695class Slots:
696
697 """Controller of the array of subprocess slots."""
698
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900699 def __init__(self, configs, options, progress):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900700 """Create a new slots controller.
701
702 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900703 configs: A list of CONFIGs to move.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900704 options: option flags.
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900705 progress: A progress indicator.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900706 """
707 self.options = options
708 self.slots = []
709 devnull = get_devnull()
710 make_cmd = get_make_cmd()
711 for i in range(options.jobs):
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900712 self.slots.append(Slot(configs, options, progress, devnull,
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900713 make_cmd))
Masahiro Yamadab6160812015-05-20 11:36:07 +0900714
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900715 def add(self, defconfig):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900716 """Add a new subprocess if a vacant slot is found.
717
718 Arguments:
719 defconfig: defconfig name to be put into.
720
721 Returns:
722 Return True on success or False on failure
723 """
724 for slot in self.slots:
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900725 if slot.add(defconfig):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900726 return True
727 return False
728
729 def available(self):
730 """Check if there is a vacant slot.
731
732 Returns:
733 Return True if at lease one vacant slot is found, False otherwise.
734 """
735 for slot in self.slots:
736 if slot.poll():
737 return True
738 return False
739
740 def empty(self):
741 """Check if all slots are vacant.
742
743 Returns:
744 Return True if all the slots are vacant, False otherwise.
745 """
746 ret = True
747 for slot in self.slots:
748 if not slot.poll():
749 ret = False
750 return ret
751
752 def show_failed_boards(self):
753 """Display all of the failed boards (defconfigs)."""
754 failed_boards = []
755
756 for slot in self.slots:
757 failed_boards += slot.get_failed_boards()
758
759 if len(failed_boards) > 0:
760 msg = [ "The following boards were not processed due to error:" ]
761 msg += failed_boards
762 for line in msg:
763 print >> sys.stderr, color_text(self.options.color,
764 COLOR_LIGHT_RED, line)
765
Joe Hershbergerdade12e2015-05-19 13:21:22 -0500766 with open('moveconfig.failed', 'w') as f:
767 for board in failed_boards:
768 f.write(board + '\n')
769
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900770def move_config(configs, options):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900771 """Move config options to defconfig files.
772
773 Arguments:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900774 configs: A list of CONFIGs to move.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900775 options: option flags
776 """
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900777 if len(configs) == 0:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900778 print 'Nothing to do. exit.'
779 sys.exit(0)
780
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900781 print 'Move %s (jobs: %d)' % (', '.join(configs), options.jobs)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900782
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500783 if options.defconfigs:
784 defconfigs = [line.strip() for line in open(options.defconfigs)]
785 for i, defconfig in enumerate(defconfigs):
786 if not defconfig.endswith('_defconfig'):
787 defconfigs[i] = defconfig + '_defconfig'
788 if not os.path.exists(os.path.join('configs', defconfigs[i])):
789 sys.exit('%s - defconfig does not exist. Stopping.' %
790 defconfigs[i])
791 else:
792 # All the defconfig files to be processed
793 defconfigs = []
794 for (dirpath, dirnames, filenames) in os.walk('configs'):
795 dirpath = dirpath[len('configs') + 1:]
796 for filename in fnmatch.filter(filenames, '*_defconfig'):
797 defconfigs.append(os.path.join(dirpath, filename))
Masahiro Yamadab6160812015-05-20 11:36:07 +0900798
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900799 progress = Progress(len(defconfigs))
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900800 slots = Slots(configs, options, progress)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900801
802 # Main loop to process defconfig files:
803 # Add a new subprocess into a vacant slot.
804 # Sleep if there is no available slot.
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900805 for defconfig in defconfigs:
806 while not slots.add(defconfig):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900807 while not slots.available():
808 # No available slot: sleep for a while
809 time.sleep(SLEEP_TIME)
810
811 # wait until all the subprocesses finish
812 while not slots.empty():
813 time.sleep(SLEEP_TIME)
814
Masahiro Yamadacefaa582016-05-19 15:51:55 +0900815 progress.show()
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500816 print ''
Masahiro Yamadab6160812015-05-20 11:36:07 +0900817 slots.show_failed_boards()
818
Masahiro Yamadab6160812015-05-20 11:36:07 +0900819def main():
820 try:
821 cpu_count = multiprocessing.cpu_count()
822 except NotImplementedError:
823 cpu_count = 1
824
825 parser = optparse.OptionParser()
826 # Add options here
827 parser.add_option('-c', '--color', action='store_true', default=False,
828 help='display the log in color')
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500829 parser.add_option('-d', '--defconfigs', type='string',
830 help='a file containing a list of defconfigs to move')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900831 parser.add_option('-n', '--dry-run', action='store_true', default=False,
832 help='perform a trial run (show log with no changes)')
833 parser.add_option('-e', '--exit-on-error', action='store_true',
834 default=False,
835 help='exit immediately on any error')
Joe Hershberger23475932015-05-19 13:21:20 -0500836 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
837 action='store_true', default=False,
838 help='only cleanup the headers')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900839 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
840 help='the number of jobs to run simultaneously')
Joe Hershberger808b63f2015-05-19 13:21:24 -0500841 parser.add_option('-v', '--verbose', action='store_true', default=False,
842 help='show any build errors as boards are built')
Masahiro Yamadab903c4e2016-05-19 15:51:58 +0900843 parser.usage += ' CONFIG ...'
Masahiro Yamadab6160812015-05-20 11:36:07 +0900844
Masahiro Yamadab903c4e2016-05-19 15:51:58 +0900845 (options, configs) = parser.parse_args()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900846
Masahiro Yamadab903c4e2016-05-19 15:51:58 +0900847 if len(configs) == 0:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900848 parser.print_usage()
849 sys.exit(1)
850
Masahiro Yamadab903c4e2016-05-19 15:51:58 +0900851 # prefix the option name with CONFIG_ if missing
852 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
853 for config in configs ]
Masahiro Yamadab6160812015-05-20 11:36:07 +0900854
Joe Hershberger23475932015-05-19 13:21:20 -0500855 check_top_directory()
856
Masahiro Yamada990e6772016-05-19 15:51:54 +0900857 check_clean_directory()
858
Masahiro Yamadac4d76eb2016-05-19 15:51:53 +0900859 update_cross_compile(options.color)
Masahiro Yamada06f76e92016-05-19 15:51:52 +0900860
Joe Hershberger23475932015-05-19 13:21:20 -0500861 if not options.cleanup_headers_only:
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900862 move_config(configs, options)
Joe Hershberger23475932015-05-19 13:21:20 -0500863
Masahiro Yamadab80a6672016-05-19 15:51:57 +0900864 cleanup_headers(configs, options.dry_run)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900865
866if __name__ == '__main__':
867 main()