blob: 496c90a9dc81dc70ff705bea9c1a727d8175ed16 [file] [log] [blame]
Masahiro Yamadab6160812015-05-20 11:36:07 +09001#!/usr/bin/env python2
2#
3# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
4#
5# SPDX-License-Identifier: GPL-2.0+
6#
7
8"""
9Move config options from headers to defconfig files.
10
11Since Kconfig was introduced to U-Boot, we have worked on moving
12config options from headers to Kconfig (defconfig).
13
14This tool intends to help this tremendous work.
15
16
17Usage
18-----
19
20This tool takes one input file. (let's say 'recipe' file here.)
21The recipe describes the list of config options you want to move.
22Each line takes the form:
23<config_name> <type> <default>
24(the fields must be separated with whitespaces.)
25
26<config_name> is the name of config option.
27
28<type> is the type of the option. It must be one of bool, tristate,
29string, int, and hex.
30
31<default> is the default value of the option. It must be appropriate
32value corresponding to the option type. It must be either y or n for
33the bool type. Tristate options can also take m (although U-Boot has
34not supported the module feature).
35
36You can add two or more lines in the recipe file, so you can move
37multiple options at once.
38
39Let's say, for example, you want to move CONFIG_CMD_USB and
40CONFIG_SYS_TEXT_BASE.
41
42The type should be bool, hex, respectively. So, the recipe file
43should look like this:
44
45 $ cat recipe
46 CONFIG_CMD_USB bool n
47 CONFIG_SYS_TEXT_BASE hex 0x00000000
48
Joe Hershberger166edec2015-05-19 13:21:17 -050049Next you must edit the Kconfig to add the menu entries for the configs
50you are moving.
51
Masahiro Yamadab6160812015-05-20 11:36:07 +090052And then run this tool giving the file name of the recipe
53
54 $ tools/moveconfig.py recipe
55
56The tool walks through all the defconfig files to move the config
57options specified by the recipe file.
58
59The log is also displayed on the terminal.
60
61Each line is printed in the format
62<defconfig_name> : <action>
63
64<defconfig_name> is the name of the defconfig
65(without the suffix _defconfig).
66
67<action> shows what the tool did for that defconfig.
68It looks like one of the followings:
69
70 - Move 'CONFIG_... '
71 This config option was moved to the defconfig
72
73 - Default value 'CONFIG_...'. Do nothing.
74 The value of this option is the same as default.
75 We do not have to add it to the defconfig.
76
77 - 'CONFIG_...' already exists in Kconfig. Do nothing.
78 This config option is already defined in Kconfig.
79 We do not need/want to touch it.
80
81 - Undefined. Do nothing.
82 This config option was not found in the config header.
83 Nothing to do.
84
85 - Failed to process. Skip.
86 An error occurred during processing this defconfig. Skipped.
87 (If -e option is passed, the tool exits immediately on error.)
88
89Finally, you will be asked, Clean up headers? [y/n]:
90
91If you say 'y' here, the unnecessary config defines are removed
92from the config headers (include/configs/*.h).
93It just uses the regex method, so you should not rely on it.
94Just in case, please do 'git diff' to see what happened.
95
96
97How does it works?
98------------------
99
100This tool runs configuration and builds include/autoconf.mk for every
101defconfig. The config options defined in Kconfig appear in the .config
102file (unless they are hidden because of unmet dependency.)
103On the other hand, the config options defined by board headers are seen
104in include/autoconf.mk. The tool looks for the specified options in both
105of them to decide the appropriate action for the options. If the option
106is found in the .config or the value is the same as the specified default,
107the option does not need to be touched. If the option is found in
108include/autoconf.mk, but not in the .config, and the value is different
109from the default, the tools adds the option to the defconfig.
110
111For faster processing, this tool handles multi-threading. It creates
112separate build directories where the out-of-tree build is run. The
113temporary build directories are automatically created and deleted as
114needed. The number of threads are chosen based on the number of the CPU
115cores of your system although you can change it via -j (--jobs) option.
116
117
118Toolchains
119----------
120
121Appropriate toolchain are necessary to generate include/autoconf.mk
122for all the architectures supported by U-Boot. Most of them are available
123at the kernel.org site, some are not provided by kernel.org.
124
125The default per-arch CROSS_COMPILE used by this tool is specified by
126the list below, CROSS_COMPILE. You may wish to update the list to
127use your own. Instead of modifying the list directly, you can give
128them via environments.
129
130
131Available options
132-----------------
133
134 -c, --color
135 Surround each portion of the log with escape sequences to display it
136 in color on the terminal.
137
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500138 -d, --defconfigs
139 Specify a file containing a list of defconfigs to move
140
Masahiro Yamadab6160812015-05-20 11:36:07 +0900141 -n, --dry-run
142 Peform a trial run that does not make any changes. It is useful to
143 see what is going to happen before one actually runs it.
144
145 -e, --exit-on-error
146 Exit immediately if Make exits with a non-zero status while processing
147 a defconfig file.
148
Joe Hershberger23475932015-05-19 13:21:20 -0500149 -H, --headers-only
150 Only cleanup the headers; skip the defconfig processing
151
Masahiro Yamadab6160812015-05-20 11:36:07 +0900152 -j, --jobs
153 Specify the number of threads to run simultaneously. If not specified,
154 the number of threads is the same as the number of CPU cores.
155
Joe Hershberger808b63f2015-05-19 13:21:24 -0500156 -v, --verbose
157 Show any build errors as boards are built
158
Masahiro Yamadab6160812015-05-20 11:36:07 +0900159To see the complete list of supported options, run
160
161 $ tools/moveconfig.py -h
162
163"""
164
165import fnmatch
166import multiprocessing
167import optparse
168import os
169import re
170import shutil
171import subprocess
172import sys
173import tempfile
174import time
175
176SHOW_GNU_MAKE = 'scripts/show-gnu-make'
177SLEEP_TIME=0.03
178
179# Here is the list of cross-tools I use.
180# Most of them are available at kernel.org
181# (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the followings:
182# arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
183# blackfin: http://sourceforge.net/projects/adi-toolchain/files/
184# nds32: http://osdk.andestech.com/packages/
185# nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
186# sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
187CROSS_COMPILE = {
188 'arc': 'arc-linux-',
189 'aarch64': 'aarch64-linux-',
190 'arm': 'arm-unknown-linux-gnueabi-',
191 'avr32': 'avr32-linux-',
192 'blackfin': 'bfin-elf-',
193 'm68k': 'm68k-linux-',
194 'microblaze': 'microblaze-linux-',
195 'mips': 'mips-linux-',
196 'nds32': 'nds32le-linux-',
197 'nios2': 'nios2-linux-gnu-',
198 'openrisc': 'or32-linux-',
199 'powerpc': 'powerpc-linux-',
200 'sh': 'sh-linux-gnu-',
201 'sparc': 'sparc-linux-',
202 'x86': 'i386-linux-'
203}
204
205STATE_IDLE = 0
206STATE_DEFCONFIG = 1
207STATE_AUTOCONF = 2
Joe Hershberger166edec2015-05-19 13:21:17 -0500208STATE_SAVEDEFCONFIG = 3
Masahiro Yamadab6160812015-05-20 11:36:07 +0900209
210ACTION_MOVE = 0
211ACTION_DEFAULT_VALUE = 1
212ACTION_ALREADY_EXIST = 2
213ACTION_UNDEFINED = 3
214
215COLOR_BLACK = '0;30'
216COLOR_RED = '0;31'
217COLOR_GREEN = '0;32'
218COLOR_BROWN = '0;33'
219COLOR_BLUE = '0;34'
220COLOR_PURPLE = '0;35'
221COLOR_CYAN = '0;36'
222COLOR_LIGHT_GRAY = '0;37'
223COLOR_DARK_GRAY = '1;30'
224COLOR_LIGHT_RED = '1;31'
225COLOR_LIGHT_GREEN = '1;32'
226COLOR_YELLOW = '1;33'
227COLOR_LIGHT_BLUE = '1;34'
228COLOR_LIGHT_PURPLE = '1;35'
229COLOR_LIGHT_CYAN = '1;36'
230COLOR_WHITE = '1;37'
231
232### helper functions ###
233def get_devnull():
234 """Get the file object of '/dev/null' device."""
235 try:
236 devnull = subprocess.DEVNULL # py3k
237 except AttributeError:
238 devnull = open(os.devnull, 'wb')
239 return devnull
240
241def check_top_directory():
242 """Exit if we are not at the top of source directory."""
243 for f in ('README', 'Licenses'):
244 if not os.path.exists(f):
245 sys.exit('Please run at the top of source directory.')
246
247def get_make_cmd():
248 """Get the command name of GNU Make.
249
250 U-Boot needs GNU Make for building, but the command name is not
251 necessarily "make". (for example, "gmake" on FreeBSD).
252 Returns the most appropriate command name on your system.
253 """
254 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
255 ret = process.communicate()
256 if process.returncode:
257 sys.exit('GNU Make not found')
258 return ret[0].rstrip()
259
260def color_text(color_enabled, color, string):
261 """Return colored string."""
262 if color_enabled:
263 return '\033[' + color + 'm' + string + '\033[0m'
264 else:
265 return string
266
267def log_msg(color_enabled, color, defconfig, msg):
268 """Return the formated line for the log."""
269 return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \
270 color_text(color_enabled, color, msg) + '\n'
271
272def update_cross_compile():
273 """Update per-arch CROSS_COMPILE via enviroment variables
274
275 The default CROSS_COMPILE values are available
276 in the CROSS_COMPILE list above.
277
278 You can override them via enviroment variables
279 CROSS_COMPILE_{ARCH}.
280
281 For example, if you want to override toolchain prefixes
282 for ARM and PowerPC, you can do as follows in your shell:
283
284 export CROSS_COMPILE_ARM=...
285 export CROSS_COMPILE_POWERPC=...
286 """
287 archs = []
288
289 for arch in os.listdir('arch'):
290 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
291 archs.append(arch)
292
293 # arm64 is a special case
294 archs.append('aarch64')
295
296 for arch in archs:
297 env = 'CROSS_COMPILE_' + arch.upper()
298 cross_compile = os.environ.get(env)
299 if cross_compile:
300 CROSS_COMPILE[arch] = cross_compile
301
302def cleanup_one_header(header_path, patterns, dry_run):
303 """Clean regex-matched lines away from a file.
304
305 Arguments:
306 header_path: path to the cleaned file.
307 patterns: list of regex patterns. Any lines matching to these
308 patterns are deleted.
309 dry_run: make no changes, but still display log.
310 """
311 with open(header_path) as f:
312 lines = f.readlines()
313
314 matched = []
315 for i, line in enumerate(lines):
316 for pattern in patterns:
317 m = pattern.search(line)
318 if m:
319 print '%s: %s: %s' % (header_path, i + 1, line),
320 matched.append(i)
321 break
322
323 if dry_run or not matched:
324 return
325
326 with open(header_path, 'w') as f:
327 for i, line in enumerate(lines):
328 if not i in matched:
329 f.write(line)
330
331def cleanup_headers(config_attrs, dry_run):
332 """Delete config defines from board headers.
333
334 Arguments:
335 config_attrs: A list of dictionaris, each of them includes the name,
336 the type, and the default value of the target config.
337 dry_run: make no changes, but still display log.
338 """
339 while True:
340 choice = raw_input('Clean up headers? [y/n]: ').lower()
341 print choice
342 if choice == 'y' or choice == 'n':
343 break
344
345 if choice == 'n':
346 return
347
348 patterns = []
349 for config_attr in config_attrs:
350 config = config_attr['config']
351 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
352 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
353
Joe Hershbergerb78ad422015-05-19 13:21:21 -0500354 for dir in 'include', 'arch', 'board':
355 for (dirpath, dirnames, filenames) in os.walk(dir):
356 for filename in filenames:
357 if not fnmatch.fnmatch(filename, '*~'):
358 cleanup_one_header(os.path.join(dirpath, filename),
359 patterns, dry_run)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900360
361### classes ###
362class KconfigParser:
363
364 """A parser of .config and include/autoconf.mk."""
365
366 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
367 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
368
369 def __init__(self, config_attrs, options, build_dir):
370 """Create a new parser.
371
372 Arguments:
373 config_attrs: A list of dictionaris, each of them includes the name,
374 the type, and the default value of the target config.
375 options: option flags.
376 build_dir: Build directory.
377 """
378 self.config_attrs = config_attrs
379 self.options = options
380 self.build_dir = build_dir
381
382 def get_cross_compile(self):
383 """Parse .config file and return CROSS_COMPILE.
384
385 Returns:
386 A string storing the compiler prefix for the architecture.
387 """
388 arch = ''
389 cpu = ''
390 dotconfig = os.path.join(self.build_dir, '.config')
391 for line in open(dotconfig):
392 m = self.re_arch.match(line)
393 if m:
394 arch = m.group(1)
395 continue
396 m = self.re_cpu.match(line)
397 if m:
398 cpu = m.group(1)
399
400 assert arch, 'Error: arch is not defined in %s' % defconfig
401
402 # fix-up for aarch64
403 if arch == 'arm' and cpu == 'armv8':
404 arch = 'aarch64'
405
406 return CROSS_COMPILE.get(arch, '')
407
Joe Hershberger166edec2015-05-19 13:21:17 -0500408 def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900409 """Parse .config, defconfig, include/autoconf.mk for one config.
410
411 This function looks for the config options in the lines from
412 defconfig, .config, and include/autoconf.mk in order to decide
413 which action should be taken for this defconfig.
414
415 Arguments:
416 config_attr: A dictionary including the name, the type,
417 and the default value of the target config.
418 defconfig_lines: lines from the original defconfig file.
Masahiro Yamadab6160812015-05-20 11:36:07 +0900419 autoconf_lines: lines from the include/autoconf.mk file.
420
421 Returns:
422 A tupple of the action for this defconfig and the line
423 matched for the config.
424 """
425 config = config_attr['config']
426 not_set = '# %s is not set' % config
427
428 if config_attr['type'] in ('bool', 'tristate') and \
429 config_attr['default'] == 'n':
430 default = not_set
431 else:
432 default = config + '=' + config_attr['default']
433
Joe Hershberger166edec2015-05-19 13:21:17 -0500434 for line in defconfig_lines:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900435 line = line.rstrip()
436 if line.startswith(config + '=') or line == not_set:
437 return (ACTION_ALREADY_EXIST, line)
438
439 if config_attr['type'] in ('bool', 'tristate'):
440 value = not_set
441 else:
442 value = '(undefined)'
443
444 for line in autoconf_lines:
445 line = line.rstrip()
446 if line.startswith(config + '='):
447 value = line
448 break
449
450 if value == default:
451 action = ACTION_DEFAULT_VALUE
452 elif value == '(undefined)':
453 action = ACTION_UNDEFINED
454 else:
455 action = ACTION_MOVE
456
457 return (action, value)
458
459 def update_defconfig(self, defconfig):
460 """Parse files for the config options and update the defconfig.
461
462 This function parses the given defconfig, the generated .config
463 and include/autoconf.mk searching the target options.
464 Move the config option(s) to the defconfig or do nothing if unneeded.
465 Also, display the log to show what happened to this defconfig.
466
467 Arguments:
468 defconfig: defconfig name.
469 """
470
471 defconfig_path = os.path.join('configs', defconfig)
472 dotconfig_path = os.path.join(self.build_dir, '.config')
473 autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk')
474 results = []
475
476 with open(defconfig_path) as f:
477 defconfig_lines = f.readlines()
478
Masahiro Yamadab6160812015-05-20 11:36:07 +0900479 with open(autoconf_path) as f:
480 autoconf_lines = f.readlines()
481
482 for config_attr in self.config_attrs:
483 result = self.parse_one_config(config_attr, defconfig_lines,
Joe Hershberger166edec2015-05-19 13:21:17 -0500484 autoconf_lines)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900485 results.append(result)
486
487 log = ''
488
489 for (action, value) in results:
490 if action == ACTION_MOVE:
491 actlog = "Move '%s'" % value
492 log_color = COLOR_LIGHT_GREEN
493 elif action == ACTION_DEFAULT_VALUE:
494 actlog = "Default value '%s'. Do nothing." % value
495 log_color = COLOR_LIGHT_BLUE
496 elif action == ACTION_ALREADY_EXIST:
497 actlog = "'%s' already defined in Kconfig. Do nothing." % value
498 log_color = COLOR_LIGHT_PURPLE
499 elif action == ACTION_UNDEFINED:
500 actlog = "Undefined. Do nothing."
501 log_color = COLOR_DARK_GRAY
502 else:
503 sys.exit("Internal Error. This should not happen.")
504
505 log += log_msg(self.options.color, log_color, defconfig, actlog)
506
507 # Some threads are running in parallel.
508 # Print log in one shot to not mix up logs from different threads.
509 print log,
510
511 if not self.options.dry_run:
Joe Hershberger166edec2015-05-19 13:21:17 -0500512 with open(dotconfig_path, 'a') as f:
Masahiro Yamadab6160812015-05-20 11:36:07 +0900513 for (action, value) in results:
514 if action == ACTION_MOVE:
515 f.write(value + '\n')
516
517 os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf'))
518 os.remove(autoconf_path)
519
520class Slot:
521
522 """A slot to store a subprocess.
523
524 Each instance of this class handles one subprocess.
525 This class is useful to control multiple threads
526 for faster processing.
527 """
528
529 def __init__(self, config_attrs, options, devnull, make_cmd):
530 """Create a new process slot.
531
532 Arguments:
533 config_attrs: A list of dictionaris, each of them includes the name,
534 the type, and the default value of the target config.
535 options: option flags.
536 devnull: A file object of '/dev/null'.
537 make_cmd: command name of GNU Make.
538 """
539 self.options = options
540 self.build_dir = tempfile.mkdtemp()
541 self.devnull = devnull
542 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
543 self.parser = KconfigParser(config_attrs, options, self.build_dir)
544 self.state = STATE_IDLE
545 self.failed_boards = []
546
547 def __del__(self):
548 """Delete the working directory
549
550 This function makes sure the temporary directory is cleaned away
551 even if Python suddenly dies due to error. It should be done in here
552 because it is guranteed the destructor is always invoked when the
553 instance of the class gets unreferenced.
554
555 If the subprocess is still running, wait until it finishes.
556 """
557 if self.state != STATE_IDLE:
558 while self.ps.poll() == None:
559 pass
560 shutil.rmtree(self.build_dir)
561
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500562 def add(self, defconfig, num, total):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900563 """Assign a new subprocess for defconfig and add it to the slot.
564
565 If the slot is vacant, create a new subprocess for processing the
566 given defconfig and add it to the slot. Just returns False if
567 the slot is occupied (i.e. the current subprocess is still running).
568
569 Arguments:
570 defconfig: defconfig name.
571
572 Returns:
573 Return True on success or False on failure
574 """
575 if self.state != STATE_IDLE:
576 return False
577 cmd = list(self.make_cmd)
578 cmd.append(defconfig)
Joe Hershberger11b02702015-05-19 13:21:23 -0500579 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
580 stderr=subprocess.PIPE)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900581 self.defconfig = defconfig
582 self.state = STATE_DEFCONFIG
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500583 self.num = num
584 self.total = total
Masahiro Yamadab6160812015-05-20 11:36:07 +0900585 return True
586
587 def poll(self):
588 """Check the status of the subprocess and handle it as needed.
589
590 Returns True if the slot is vacant (i.e. in idle state).
591 If the configuration is successfully finished, assign a new
592 subprocess to build include/autoconf.mk.
593 If include/autoconf.mk is generated, invoke the parser to
594 parse the .config and the include/autoconf.mk, and then set the
595 slot back to the idle state.
596
597 Returns:
598 Return True if the subprocess is terminated, False otherwise
599 """
600 if self.state == STATE_IDLE:
601 return True
602
603 if self.ps.poll() == None:
604 return False
605
606 if self.ps.poll() != 0:
Joe Hershberger11b02702015-05-19 13:21:23 -0500607 errmsg = 'Failed to process.'
608 errout = self.ps.stderr.read()
609 if errout.find('gcc: command not found') != -1:
610 errmsg = 'Compiler not found ('
611 errmsg += color_text(self.options.color, COLOR_YELLOW,
612 self.cross_compile)
613 errmsg += color_text(self.options.color, COLOR_LIGHT_RED,
614 ')')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900615 print >> sys.stderr, log_msg(self.options.color,
616 COLOR_LIGHT_RED,
617 self.defconfig,
Joe Hershberger11b02702015-05-19 13:21:23 -0500618 errmsg),
Joe Hershberger808b63f2015-05-19 13:21:24 -0500619 if self.options.verbose:
620 print >> sys.stderr, color_text(self.options.color,
621 COLOR_LIGHT_CYAN, errout)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900622 if self.options.exit_on_error:
623 sys.exit("Exit on error.")
624 else:
625 # If --exit-on-error flag is not set,
626 # skip this board and continue.
627 # Record the failed board.
628 self.failed_boards.append(self.defconfig)
629 self.state = STATE_IDLE
630 return True
631
632 if self.state == STATE_AUTOCONF:
633 self.parser.update_defconfig(self.defconfig)
Joe Hershberger166edec2015-05-19 13:21:17 -0500634
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500635 print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
636 sys.stdout.flush()
637
Joe Hershberger166edec2015-05-19 13:21:17 -0500638 """Save off the defconfig in a consistent way"""
639 cmd = list(self.make_cmd)
640 cmd.append('savedefconfig')
641 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
Joe Hershberger11b02702015-05-19 13:21:23 -0500642 stderr=subprocess.PIPE)
Joe Hershberger166edec2015-05-19 13:21:17 -0500643 self.state = STATE_SAVEDEFCONFIG
644 return False
645
646 if self.state == STATE_SAVEDEFCONFIG:
647 defconfig_path = os.path.join(self.build_dir, 'defconfig')
648 shutil.move(defconfig_path,
649 os.path.join('configs', self.defconfig))
Masahiro Yamadab6160812015-05-20 11:36:07 +0900650 self.state = STATE_IDLE
651 return True
652
Joe Hershberger11b02702015-05-19 13:21:23 -0500653 self.cross_compile = self.parser.get_cross_compile()
Masahiro Yamadab6160812015-05-20 11:36:07 +0900654 cmd = list(self.make_cmd)
Joe Hershberger11b02702015-05-19 13:21:23 -0500655 if self.cross_compile:
656 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
Joe Hershberger765442b2015-05-19 13:21:18 -0500657 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900658 cmd.append('include/config/auto.conf')
Joe Hershberger11b02702015-05-19 13:21:23 -0500659 """This will be screen-scraped, so be sure the expected text will be
660 returned consistently on every machine by setting LANG=C"""
661 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
662 env=dict(os.environ, LANG='C'),
663 stderr=subprocess.PIPE)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900664 self.state = STATE_AUTOCONF
665 return False
666
667 def get_failed_boards(self):
668 """Returns a list of failed boards (defconfigs) in this slot.
669 """
670 return self.failed_boards
671
672class Slots:
673
674 """Controller of the array of subprocess slots."""
675
676 def __init__(self, config_attrs, options):
677 """Create a new slots controller.
678
679 Arguments:
680 config_attrs: A list of dictionaris containing the name, the type,
681 and the default value of the target CONFIG.
682 options: option flags.
683 """
684 self.options = options
685 self.slots = []
686 devnull = get_devnull()
687 make_cmd = get_make_cmd()
688 for i in range(options.jobs):
689 self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
690
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500691 def add(self, defconfig, num, total):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900692 """Add a new subprocess if a vacant slot is found.
693
694 Arguments:
695 defconfig: defconfig name to be put into.
696
697 Returns:
698 Return True on success or False on failure
699 """
700 for slot in self.slots:
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500701 if slot.add(defconfig, num, total):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900702 return True
703 return False
704
705 def available(self):
706 """Check if there is a vacant slot.
707
708 Returns:
709 Return True if at lease one vacant slot is found, False otherwise.
710 """
711 for slot in self.slots:
712 if slot.poll():
713 return True
714 return False
715
716 def empty(self):
717 """Check if all slots are vacant.
718
719 Returns:
720 Return True if all the slots are vacant, False otherwise.
721 """
722 ret = True
723 for slot in self.slots:
724 if not slot.poll():
725 ret = False
726 return ret
727
728 def show_failed_boards(self):
729 """Display all of the failed boards (defconfigs)."""
730 failed_boards = []
731
732 for slot in self.slots:
733 failed_boards += slot.get_failed_boards()
734
735 if len(failed_boards) > 0:
736 msg = [ "The following boards were not processed due to error:" ]
737 msg += failed_boards
738 for line in msg:
739 print >> sys.stderr, color_text(self.options.color,
740 COLOR_LIGHT_RED, line)
741
Joe Hershbergerdade12e2015-05-19 13:21:22 -0500742 with open('moveconfig.failed', 'w') as f:
743 for board in failed_boards:
744 f.write(board + '\n')
745
Masahiro Yamadab6160812015-05-20 11:36:07 +0900746def move_config(config_attrs, options):
747 """Move config options to defconfig files.
748
749 Arguments:
750 config_attrs: A list of dictionaris, each of them includes the name,
751 the type, and the default value of the target config.
752 options: option flags
753 """
Masahiro Yamadab6160812015-05-20 11:36:07 +0900754 if len(config_attrs) == 0:
755 print 'Nothing to do. exit.'
756 sys.exit(0)
757
758 print 'Move the following CONFIG options (jobs: %d)' % options.jobs
759 for config_attr in config_attrs:
760 print ' %s (type: %s, default: %s)' % (config_attr['config'],
761 config_attr['type'],
762 config_attr['default'])
763
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500764 if options.defconfigs:
765 defconfigs = [line.strip() for line in open(options.defconfigs)]
766 for i, defconfig in enumerate(defconfigs):
767 if not defconfig.endswith('_defconfig'):
768 defconfigs[i] = defconfig + '_defconfig'
769 if not os.path.exists(os.path.join('configs', defconfigs[i])):
770 sys.exit('%s - defconfig does not exist. Stopping.' %
771 defconfigs[i])
772 else:
773 # All the defconfig files to be processed
774 defconfigs = []
775 for (dirpath, dirnames, filenames) in os.walk('configs'):
776 dirpath = dirpath[len('configs') + 1:]
777 for filename in fnmatch.filter(filenames, '*_defconfig'):
778 defconfigs.append(os.path.join(dirpath, filename))
Masahiro Yamadab6160812015-05-20 11:36:07 +0900779
780 slots = Slots(config_attrs, options)
781
782 # Main loop to process defconfig files:
783 # Add a new subprocess into a vacant slot.
784 # Sleep if there is no available slot.
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500785 for i, defconfig in enumerate(defconfigs):
786 while not slots.add(defconfig, i, len(defconfigs)):
Masahiro Yamadab6160812015-05-20 11:36:07 +0900787 while not slots.available():
788 # No available slot: sleep for a while
789 time.sleep(SLEEP_TIME)
790
791 # wait until all the subprocesses finish
792 while not slots.empty():
793 time.sleep(SLEEP_TIME)
794
Joe Hershberger3fa1ab72015-05-19 13:21:25 -0500795 print ''
Masahiro Yamadab6160812015-05-20 11:36:07 +0900796 slots.show_failed_boards()
797
Masahiro Yamadab6160812015-05-20 11:36:07 +0900798def bad_recipe(filename, linenum, msg):
799 """Print error message with the file name and the line number and exit."""
800 sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
801
802def parse_recipe(filename):
803 """Parse the recipe file and retrieve the config attributes.
804
805 This function parses the given recipe file and gets the name,
806 the type, and the default value of the target config options.
807
808 Arguments:
809 filename: path to file to be parsed.
810 Returns:
811 A list of dictionaris, each of them includes the name,
812 the type, and the default value of the target config.
813 """
814 config_attrs = []
815 linenum = 1
816
817 for line in open(filename):
818 tokens = line.split()
819 if len(tokens) != 3:
820 bad_recipe(filename, linenum,
821 "%d fields in this line. Each line must contain 3 fields"
822 % len(tokens))
823
824 (config, type, default) = tokens
825
826 # prefix the option name with CONFIG_ if missing
827 if not config.startswith('CONFIG_'):
828 config = 'CONFIG_' + config
829
830 # sanity check of default values
831 if type == 'bool':
832 if not default in ('y', 'n'):
833 bad_recipe(filename, linenum,
834 "default for bool type must be either y or n")
835 elif type == 'tristate':
836 if not default in ('y', 'm', 'n'):
837 bad_recipe(filename, linenum,
838 "default for tristate type must be y, m, or n")
839 elif type == 'string':
840 if default[0] != '"' or default[-1] != '"':
841 bad_recipe(filename, linenum,
842 "default for string type must be surrounded by double-quotations")
843 elif type == 'int':
844 try:
845 int(default)
846 except:
847 bad_recipe(filename, linenum,
848 "type is int, but default value is not decimal")
849 elif type == 'hex':
850 if len(default) < 2 or default[:2] != '0x':
851 bad_recipe(filename, linenum,
852 "default for hex type must be prefixed with 0x")
853 try:
854 int(default, 16)
855 except:
856 bad_recipe(filename, linenum,
857 "type is hex, but default value is not hexadecimal")
858 else:
859 bad_recipe(filename, linenum,
860 "unsupported type '%s'. type must be one of bool, tristate, string, int, hex"
861 % type)
862
863 config_attrs.append({'config': config, 'type': type, 'default': default})
864 linenum += 1
865
866 return config_attrs
867
868def main():
869 try:
870 cpu_count = multiprocessing.cpu_count()
871 except NotImplementedError:
872 cpu_count = 1
873
874 parser = optparse.OptionParser()
875 # Add options here
876 parser.add_option('-c', '--color', action='store_true', default=False,
877 help='display the log in color')
Joe Hershbergerc6e043a2015-05-19 13:21:19 -0500878 parser.add_option('-d', '--defconfigs', type='string',
879 help='a file containing a list of defconfigs to move')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900880 parser.add_option('-n', '--dry-run', action='store_true', default=False,
881 help='perform a trial run (show log with no changes)')
882 parser.add_option('-e', '--exit-on-error', action='store_true',
883 default=False,
884 help='exit immediately on any error')
Joe Hershberger23475932015-05-19 13:21:20 -0500885 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
886 action='store_true', default=False,
887 help='only cleanup the headers')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900888 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
889 help='the number of jobs to run simultaneously')
Joe Hershberger808b63f2015-05-19 13:21:24 -0500890 parser.add_option('-v', '--verbose', action='store_true', default=False,
891 help='show any build errors as boards are built')
Masahiro Yamadab6160812015-05-20 11:36:07 +0900892 parser.usage += ' recipe_file\n\n' + \
893 'The recipe_file should describe config options you want to move.\n' + \
894 'Each line should contain config_name, type, default_value\n\n' + \
895 'Example:\n' + \
896 'CONFIG_FOO bool n\n' + \
897 'CONFIG_BAR int 100\n' + \
898 'CONFIG_BAZ string "hello"\n'
899
900 (options, args) = parser.parse_args()
901
902 if len(args) != 1:
903 parser.print_usage()
904 sys.exit(1)
905
906 config_attrs = parse_recipe(args[0])
907
908 update_cross_compile()
909
Joe Hershberger23475932015-05-19 13:21:20 -0500910 check_top_directory()
911
912 if not options.cleanup_headers_only:
913 move_config(config_attrs, options)
914
915 cleanup_headers(config_attrs, options.dry_run)
Masahiro Yamadab6160812015-05-20 11:36:07 +0900916
917if __name__ == '__main__':
918 main()