3 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5 # SPDX-License-Identifier: GPL-2.0+
9 Move config options from headers to defconfig files.
11 Since Kconfig was introduced to U-Boot, we have worked on moving
12 config options from headers to Kconfig (defconfig).
14 This tool intends to help this tremendous work.
20 First, you must edit the Kconfig to add the menu entries for the configs
23 And then run this tool giving CONFIG names you want to move.
24 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25 simply type as follows:
27 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
29 The tool walks through all the defconfig files and move the given CONFIGs.
31 The log is also displayed on the terminal.
33 The log is printed for each defconfig as follows:
41 <defconfig_name> is the name of the defconfig.
43 <action*> shows what the tool did for that defconfig.
44 It looks like one of the following:
47 This config option was moved to the defconfig
49 - CONFIG_... is not defined in Kconfig. Do nothing.
50 The entry for this CONFIG was not found in Kconfig. The option is not
51 defined in the config header, either. So, this case can be just skipped.
53 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
54 This option is defined in the config header, but its entry was not found
56 There are two common cases:
57 - You forgot to create an entry for the CONFIG before running
58 this tool, or made a typo in a CONFIG passed to this tool.
59 - The entry was hidden due to unmet 'depends on'.
60 The tool does not know if the result is reasonable, so please check it
63 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
64 The define in the config header matched the one in Kconfig.
65 We do not need to touch it.
67 - Compiler is missing. Do nothing.
68 The compiler specified for this architecture was not found
69 in your PATH environment.
70 (If -e option is passed, the tool exits immediately.)
73 An error occurred during processing this defconfig. Skipped.
74 (If -e option is passed, the tool exits immediately on error.)
76 Finally, you will be asked, Clean up headers? [y/n]:
78 If you say 'y' here, the unnecessary config defines are removed
79 from the config headers (include/configs/*.h).
80 It just uses the regex method, so you should not rely on it.
81 Just in case, please do 'git diff' to see what happened.
87 This tool runs configuration and builds include/autoconf.mk for every
88 defconfig. The config options defined in Kconfig appear in the .config
89 file (unless they are hidden because of unmet dependency.)
90 On the other hand, the config options defined by board headers are seen
91 in include/autoconf.mk. The tool looks for the specified options in both
92 of them to decide the appropriate action for the options. If the given
93 config option is found in the .config, but its value does not match the
94 one from the board header, the config option in the .config is replaced
95 with the define in the board header. Then, the .config is synced by
96 "make savedefconfig" and the defconfig is updated with it.
98 For faster processing, this tool handles multi-threading. It creates
99 separate build directories where the out-of-tree build is run. The
100 temporary build directories are automatically created and deleted as
101 needed. The number of threads are chosen based on the number of the CPU
102 cores of your system although you can change it via -j (--jobs) option.
108 Appropriate toolchain are necessary to generate include/autoconf.mk
109 for all the architectures supported by U-Boot. Most of them are available
110 at the kernel.org site, some are not provided by kernel.org.
112 The default per-arch CROSS_COMPILE used by this tool is specified by
113 the list below, CROSS_COMPILE. You may wish to update the list to
114 use your own. Instead of modifying the list directly, you can give
115 them via environments.
121 To sync only X86 defconfigs:
123 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*)
127 grep -l X86 configs/* | ./tools/moveconfig.py -s -d -
129 To process CONFIG_CMD_FPGAD only for a subset of configs based on path match:
131 ls configs/{hrcon*,iocon*,strider*} | \
132 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
139 Surround each portion of the log with escape sequences to display it
140 in color on the terminal.
143 Create a git commit with the changes when the operation is complete. A
144 standard commit message is used which may need to be edited.
147 Specify a file containing a list of defconfigs to move. The defconfig
148 files can be given with shell-style wildcards. Use '-' to read from stdin.
151 Perform a trial run that does not make any changes. It is useful to
152 see what is going to happen before one actually runs it.
155 Exit immediately if Make exits with a non-zero status while processing
159 Do "make savedefconfig" forcibly for all the defconfig files.
160 If not specified, "make savedefconfig" only occurs for cases
161 where at least one CONFIG was moved.
164 Look for moved config options in spl/include/autoconf.mk instead of
165 include/autoconf.mk. This is useful for moving options for SPL build
166 because SPL related options (mostly prefixed with CONFIG_SPL_) are
167 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
170 Only cleanup the headers; skip the defconfig processing
173 Specify the number of threads to run simultaneously. If not specified,
174 the number of threads is the same as the number of CPU cores.
177 Specify the git ref to clone for building the autoconf.mk. If unspecified
178 use the CWD. This is useful for when changes to the Kconfig affect the
179 default values and you want to capture the state of the defconfig from
180 before that change was in effect. If in doubt, specify a ref pre-Kconfig
181 changes (use HEAD if Kconfig changes are not committed). Worst case it will
182 take a bit longer to run, but will always do the right thing.
185 Show any build errors as boards are built
188 Instead of prompting, automatically go ahead with all operations. This
189 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
192 To see the complete list of supported options, run
194 $ tools/moveconfig.py -h
203 import multiprocessing
213 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
216 # Here is the list of cross-tools I use.
217 # Most of them are available at kernel.org
218 # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
219 # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
220 # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
221 # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
222 # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
225 'aarch64': 'aarch64-linux-',
226 'arm': 'arm-unknown-linux-gnueabi-',
227 'm68k': 'm68k-linux-',
228 'microblaze': 'microblaze-linux-',
229 'mips': 'mips-linux-',
230 'nds32': 'nds32le-linux-',
231 'nios2': 'nios2-linux-gnu-',
232 'powerpc': 'powerpc-linux-',
233 'sh': 'sh-linux-gnu-',
234 'x86': 'i386-linux-',
235 'xtensa': 'xtensa-linux-'
241 STATE_SAVEDEFCONFIG = 3
245 ACTION_NO_ENTRY_WARN = 2
253 COLOR_PURPLE = '0;35'
255 COLOR_LIGHT_GRAY = '0;37'
256 COLOR_DARK_GRAY = '1;30'
257 COLOR_LIGHT_RED = '1;31'
258 COLOR_LIGHT_GREEN = '1;32'
259 COLOR_YELLOW = '1;33'
260 COLOR_LIGHT_BLUE = '1;34'
261 COLOR_LIGHT_PURPLE = '1;35'
262 COLOR_LIGHT_CYAN = '1;36'
265 ### helper functions ###
267 """Get the file object of '/dev/null' device."""
269 devnull = subprocess.DEVNULL # py3k
270 except AttributeError:
271 devnull = open(os.devnull, 'wb')
274 def check_top_directory():
275 """Exit if we are not at the top of source directory."""
276 for f in ('README', 'Licenses'):
277 if not os.path.exists(f):
278 sys.exit('Please run at the top of source directory.')
280 def check_clean_directory():
281 """Exit if the source tree is not clean."""
282 for f in ('.config', 'include/config'):
283 if os.path.exists(f):
284 sys.exit("source tree is not clean, please run 'make mrproper'")
287 """Get the command name of GNU Make.
289 U-Boot needs GNU Make for building, but the command name is not
290 necessarily "make". (for example, "gmake" on FreeBSD).
291 Returns the most appropriate command name on your system.
293 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
294 ret = process.communicate()
295 if process.returncode:
296 sys.exit('GNU Make not found')
297 return ret[0].rstrip()
299 def get_matched_defconfig(line):
300 """Get the defconfig files that match a pattern
303 line: Path or filename to match, e.g. 'configs/snow_defconfig' or
304 'k2*_defconfig'. If no directory is provided, 'configs/' is
308 a list of matching defconfig files
310 dirname = os.path.dirname(line)
314 pattern = os.path.join('configs', line)
315 return glob.glob(pattern) + glob.glob(pattern + '_defconfig')
317 def get_matched_defconfigs(defconfigs_file):
318 """Get all the defconfig files that match the patterns in a file.
321 defconfigs_file: File containing a list of defconfigs to process, or
322 '-' to read the list from stdin
325 A list of paths to defconfig files, with no duplicates
328 if defconfigs_file == '-':
330 defconfigs_file = 'stdin'
332 fd = open(defconfigs_file)
333 for i, line in enumerate(fd):
336 continue # skip blank lines silently
337 matched = get_matched_defconfig(line)
339 print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
340 (defconfigs_file, i + 1, line)
342 defconfigs += matched
344 # use set() to drop multiple matching
345 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ]
347 def get_all_defconfigs():
348 """Get all the defconfig files under the configs/ directory."""
350 for (dirpath, dirnames, filenames) in os.walk('configs'):
351 dirpath = dirpath[len('configs') + 1:]
352 for filename in fnmatch.filter(filenames, '*_defconfig'):
353 defconfigs.append(os.path.join(dirpath, filename))
357 def color_text(color_enabled, color, string):
358 """Return colored string."""
360 # LF should not be surrounded by the escape sequence.
361 # Otherwise, additional whitespace or line-feed might be printed.
362 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
363 for s in string.split('\n') ])
367 def show_diff(a, b, file_path, color_enabled):
368 """Show unidified diff.
371 a: A list of lines (before)
372 b: A list of lines (after)
373 file_path: Path to the file
374 color_enabled: Display the diff in color
377 diff = difflib.unified_diff(a, b,
378 fromfile=os.path.join('a', file_path),
379 tofile=os.path.join('b', file_path))
382 if line[0] == '-' and line[1] != '-':
383 print color_text(color_enabled, COLOR_RED, line),
384 elif line[0] == '+' and line[1] != '+':
385 print color_text(color_enabled, COLOR_GREEN, line),
389 def update_cross_compile(color_enabled):
390 """Update per-arch CROSS_COMPILE via environment variables
392 The default CROSS_COMPILE values are available
393 in the CROSS_COMPILE list above.
395 You can override them via environment variables
396 CROSS_COMPILE_{ARCH}.
398 For example, if you want to override toolchain prefixes
399 for ARM and PowerPC, you can do as follows in your shell:
401 export CROSS_COMPILE_ARM=...
402 export CROSS_COMPILE_POWERPC=...
404 Then, this function checks if specified compilers really exist in your
409 for arch in os.listdir('arch'):
410 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
413 # arm64 is a special case
414 archs.append('aarch64')
417 env = 'CROSS_COMPILE_' + arch.upper()
418 cross_compile = os.environ.get(env)
419 if not cross_compile:
420 cross_compile = CROSS_COMPILE.get(arch, '')
422 for path in os.environ["PATH"].split(os.pathsep):
423 gcc_path = os.path.join(path, cross_compile + 'gcc')
424 if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
427 print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
428 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
429 % (cross_compile, arch))
432 CROSS_COMPILE[arch] = cross_compile
434 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
436 """Extend matched lines if desired patterns are found before/after already
440 lines: A list of lines handled.
441 matched: A list of line numbers that have been already matched.
442 (will be updated by this function)
443 pre_patterns: A list of regular expression that should be matched as
445 post_patterns: A list of regular expression that should be matched as
447 extend_pre: Add the line number of matched preamble to the matched list.
448 extend_post: Add the line number of matched postamble to the matched list.
450 extended_matched = []
463 for p in pre_patterns:
464 if p.search(lines[i - 1]):
470 for p in post_patterns:
471 if p.search(lines[j]):
478 extended_matched.append(i - 1)
480 extended_matched.append(j)
482 matched += extended_matched
485 def confirm(options, prompt):
488 choice = raw_input('{} [y/n]: '.format(prompt))
489 choice = choice.lower()
491 if choice == 'y' or choice == 'n':
499 def cleanup_one_header(header_path, patterns, options):
500 """Clean regex-matched lines away from a file.
503 header_path: path to the cleaned file.
504 patterns: list of regex patterns. Any lines matching to these
505 patterns are deleted.
506 options: option flags.
508 with open(header_path) as f:
509 lines = f.readlines()
512 for i, line in enumerate(lines):
513 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
516 for pattern in patterns:
517 if pattern.search(line):
524 # remove empty #ifdef ... #endif, successive blank lines
525 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
526 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
527 pattern_endif = re.compile(r'#\s*endif\W') # #endif
528 pattern_blank = re.compile(r'^\s*$') # empty line
531 old_matched = copy.copy(matched)
532 extend_matched_lines(lines, matched, [pattern_if],
533 [pattern_endif], True, True)
534 extend_matched_lines(lines, matched, [pattern_elif],
535 [pattern_elif, pattern_endif], True, False)
536 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
537 [pattern_blank], False, True)
538 extend_matched_lines(lines, matched, [pattern_blank],
539 [pattern_elif, pattern_endif], True, False)
540 extend_matched_lines(lines, matched, [pattern_blank],
541 [pattern_blank], True, False)
542 if matched == old_matched:
545 tolines = copy.copy(lines)
547 for i in reversed(matched):
550 show_diff(lines, tolines, header_path, options.color)
555 with open(header_path, 'w') as f:
559 def cleanup_headers(configs, options):
560 """Delete config defines from board headers.
563 configs: A list of CONFIGs to remove.
564 options: option flags.
566 if not confirm(options, 'Clean up headers?'):
570 for config in configs:
571 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
572 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
574 for dir in 'include', 'arch', 'board':
575 for (dirpath, dirnames, filenames) in os.walk(dir):
576 if dirpath == os.path.join('include', 'generated'):
578 for filename in filenames:
579 if not fnmatch.fnmatch(filename, '*~'):
580 cleanup_one_header(os.path.join(dirpath, filename),
583 def cleanup_one_extra_option(defconfig_path, configs, options):
584 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
587 defconfig_path: path to the cleaned defconfig file.
588 configs: A list of CONFIGs to remove.
589 options: option flags.
592 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
595 with open(defconfig_path) as f:
596 lines = f.readlines()
598 for i, line in enumerate(lines):
599 if line.startswith(start) and line.endswith(end):
602 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
605 old_tokens = line[len(start):-len(end)].split(',')
608 for token in old_tokens:
609 pos = token.find('=')
610 if not (token[:pos] if pos >= 0 else token) in configs:
611 new_tokens.append(token)
613 if new_tokens == old_tokens:
616 tolines = copy.copy(lines)
619 tolines[i] = start + ','.join(new_tokens) + end
623 show_diff(lines, tolines, defconfig_path, options.color)
628 with open(defconfig_path, 'w') as f:
632 def cleanup_extra_options(configs, options):
633 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
636 configs: A list of CONFIGs to remove.
637 options: option flags.
639 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
642 configs = [ config[len('CONFIG_'):] for config in configs ]
644 defconfigs = get_all_defconfigs()
646 for defconfig in defconfigs:
647 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
650 def cleanup_whitelist(configs, options):
651 """Delete config whitelist entries
654 configs: A list of CONFIGs to remove.
655 options: option flags.
657 if not confirm(options, 'Clean up whitelist entries?'):
660 with open(os.path.join('scripts', 'config_whitelist.txt')) as f:
661 lines = f.readlines()
663 lines = [x for x in lines if x.strip() not in configs]
665 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f:
666 f.write(''.join(lines))
668 def find_matching(patterns, line):
674 def cleanup_readme(configs, options):
675 """Delete config description in README
678 configs: A list of CONFIGs to remove.
679 options: option flags.
681 if not confirm(options, 'Clean up README?'):
685 for config in configs:
686 patterns.append(re.compile(r'^\s+%s' % config))
688 with open('README') as f:
689 lines = f.readlines()
695 found = find_matching(patterns, line)
699 if found and re.search(r'^\s+CONFIG', line):
703 newlines.append(line)
705 with open('README', 'w') as f:
706 f.write(''.join(newlines))
712 """Progress Indicator"""
714 def __init__(self, total):
715 """Create a new progress indicator.
718 total: A number of defconfig files to process.
724 """Increment the number of processed defconfig files."""
729 """Display the progress."""
730 print ' %d defconfigs out of %d\r' % (self.current, self.total),
735 """A parser of .config and include/autoconf.mk."""
737 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
738 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
740 def __init__(self, configs, options, build_dir):
741 """Create a new parser.
744 configs: A list of CONFIGs to move.
745 options: option flags.
746 build_dir: Build directory.
748 self.configs = configs
749 self.options = options
750 self.dotconfig = os.path.join(build_dir, '.config')
751 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
752 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
754 self.config_autoconf = os.path.join(build_dir, 'include', 'config',
756 self.defconfig = os.path.join(build_dir, 'defconfig')
758 def get_cross_compile(self):
759 """Parse .config file and return CROSS_COMPILE.
762 A string storing the compiler prefix for the architecture.
763 Return a NULL string for architectures that do not require
764 compiler prefix (Sandbox and native build is the case).
765 Return None if the specified compiler is missing in your PATH.
766 Caller should distinguish '' and None.
770 for line in open(self.dotconfig):
771 m = self.re_arch.match(line)
775 m = self.re_cpu.match(line)
783 if arch == 'arm' and cpu == 'armv8':
786 return CROSS_COMPILE.get(arch, None)
788 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
789 """Parse .config, defconfig, include/autoconf.mk for one config.
791 This function looks for the config options in the lines from
792 defconfig, .config, and include/autoconf.mk in order to decide
793 which action should be taken for this defconfig.
796 config: CONFIG name to parse.
797 dotconfig_lines: lines from the .config file.
798 autoconf_lines: lines from the include/autoconf.mk file.
801 A tupple of the action for this defconfig and the line
802 matched for the config.
804 not_set = '# %s is not set' % config
806 for line in autoconf_lines:
808 if line.startswith(config + '='):
814 for line in dotconfig_lines:
816 if line.startswith(config + '=') or line == not_set:
820 if new_val == not_set:
821 return (ACTION_NO_ENTRY, config)
823 return (ACTION_NO_ENTRY_WARN, config)
825 # If this CONFIG is neither bool nor trisate
826 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
827 # tools/scripts/define2mk.sed changes '1' to 'y'.
828 # This is a problem if the CONFIG is int type.
829 # Check the type in Kconfig and handle it correctly.
830 if new_val[-2:] == '=y':
831 new_val = new_val[:-1] + '1'
833 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
836 def update_dotconfig(self):
837 """Parse files for the config options and update the .config.
839 This function parses the generated .config and include/autoconf.mk
840 searching the target options.
841 Move the config option(s) to the .config as needed.
844 defconfig: defconfig name.
847 Return a tuple of (updated flag, log string).
848 The "updated flag" is True if the .config was updated, False
849 otherwise. The "log string" shows what happend to the .config.
855 rm_files = [self.config_autoconf, self.autoconf]
858 if os.path.exists(self.spl_autoconf):
859 autoconf_path = self.spl_autoconf
860 rm_files.append(self.spl_autoconf)
864 return (updated, suspicious,
865 color_text(self.options.color, COLOR_BROWN,
866 "SPL is not enabled. Skipped.") + '\n')
868 autoconf_path = self.autoconf
870 with open(self.dotconfig) as f:
871 dotconfig_lines = f.readlines()
873 with open(autoconf_path) as f:
874 autoconf_lines = f.readlines()
876 for config in self.configs:
877 result = self.parse_one_config(config, dotconfig_lines,
879 results.append(result)
883 for (action, value) in results:
884 if action == ACTION_MOVE:
885 actlog = "Move '%s'" % value
886 log_color = COLOR_LIGHT_GREEN
887 elif action == ACTION_NO_ENTRY:
888 actlog = "%s is not defined in Kconfig. Do nothing." % value
889 log_color = COLOR_LIGHT_BLUE
890 elif action == ACTION_NO_ENTRY_WARN:
891 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
892 log_color = COLOR_YELLOW
894 elif action == ACTION_NO_CHANGE:
895 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
897 log_color = COLOR_LIGHT_PURPLE
898 elif action == ACTION_SPL_NOT_EXIST:
899 actlog = "SPL is not enabled for this defconfig. Skip."
900 log_color = COLOR_PURPLE
902 sys.exit("Internal Error. This should not happen.")
904 log += color_text(self.options.color, log_color, actlog) + '\n'
906 with open(self.dotconfig, 'a') as f:
907 for (action, value) in results:
908 if action == ACTION_MOVE:
909 f.write(value + '\n')
912 self.results = results
916 return (updated, suspicious, log)
918 def check_defconfig(self):
919 """Check the defconfig after savedefconfig
922 Return additional log if moved CONFIGs were removed again by
923 'make savedefconfig'.
928 with open(self.defconfig) as f:
929 defconfig_lines = f.readlines()
931 for (action, value) in self.results:
932 if action != ACTION_MOVE:
934 if not value + '\n' in defconfig_lines:
935 log += color_text(self.options.color, COLOR_YELLOW,
936 "'%s' was removed by savedefconfig.\n" %
943 """A slot to store a subprocess.
945 Each instance of this class handles one subprocess.
946 This class is useful to control multiple threads
947 for faster processing.
950 def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir):
951 """Create a new process slot.
954 configs: A list of CONFIGs to move.
955 options: option flags.
956 progress: A progress indicator.
957 devnull: A file object of '/dev/null'.
958 make_cmd: command name of GNU Make.
959 reference_src_dir: Determine the true starting config state from this
962 self.options = options
963 self.progress = progress
964 self.build_dir = tempfile.mkdtemp()
965 self.devnull = devnull
966 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
967 self.reference_src_dir = reference_src_dir
968 self.parser = KconfigParser(configs, options, self.build_dir)
969 self.state = STATE_IDLE
970 self.failed_boards = set()
971 self.suspicious_boards = set()
974 """Delete the working directory
976 This function makes sure the temporary directory is cleaned away
977 even if Python suddenly dies due to error. It should be done in here
978 because it is guaranteed the destructor is always invoked when the
979 instance of the class gets unreferenced.
981 If the subprocess is still running, wait until it finishes.
983 if self.state != STATE_IDLE:
984 while self.ps.poll() == None:
986 shutil.rmtree(self.build_dir)
988 def add(self, defconfig):
989 """Assign a new subprocess for defconfig and add it to the slot.
991 If the slot is vacant, create a new subprocess for processing the
992 given defconfig and add it to the slot. Just returns False if
993 the slot is occupied (i.e. the current subprocess is still running).
996 defconfig: defconfig name.
999 Return True on success or False on failure
1001 if self.state != STATE_IDLE:
1004 self.defconfig = defconfig
1006 self.current_src_dir = self.reference_src_dir
1011 """Check the status of the subprocess and handle it as needed.
1013 Returns True if the slot is vacant (i.e. in idle state).
1014 If the configuration is successfully finished, assign a new
1015 subprocess to build include/autoconf.mk.
1016 If include/autoconf.mk is generated, invoke the parser to
1017 parse the .config and the include/autoconf.mk, moving
1018 config options to the .config as needed.
1019 If the .config was updated, run "make savedefconfig" to sync
1020 it, update the original defconfig, and then set the slot back
1024 Return True if the subprocess is terminated, False otherwise
1026 if self.state == STATE_IDLE:
1029 if self.ps.poll() == None:
1032 if self.ps.poll() != 0:
1034 elif self.state == STATE_DEFCONFIG:
1035 if self.reference_src_dir and not self.current_src_dir:
1036 self.do_savedefconfig()
1039 elif self.state == STATE_AUTOCONF:
1040 if self.current_src_dir:
1041 self.current_src_dir = None
1044 self.do_savedefconfig()
1045 elif self.state == STATE_SAVEDEFCONFIG:
1046 self.update_defconfig()
1048 sys.exit("Internal Error. This should not happen.")
1050 return True if self.state == STATE_IDLE else False
1052 def handle_error(self):
1053 """Handle error cases."""
1055 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
1056 "Failed to process.\n")
1057 if self.options.verbose:
1058 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
1059 self.ps.stderr.read())
1062 def do_defconfig(self):
1063 """Run 'make <board>_defconfig' to create the .config file."""
1065 cmd = list(self.make_cmd)
1066 cmd.append(self.defconfig)
1067 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1068 stderr=subprocess.PIPE,
1069 cwd=self.current_src_dir)
1070 self.state = STATE_DEFCONFIG
1072 def do_autoconf(self):
1073 """Run 'make include/config/auto.conf'."""
1075 self.cross_compile = self.parser.get_cross_compile()
1076 if self.cross_compile is None:
1077 self.log += color_text(self.options.color, COLOR_YELLOW,
1078 "Compiler is missing. Do nothing.\n")
1082 cmd = list(self.make_cmd)
1083 if self.cross_compile:
1084 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
1085 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
1086 cmd.append('include/config/auto.conf')
1087 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1088 stderr=subprocess.PIPE,
1089 cwd=self.current_src_dir)
1090 self.state = STATE_AUTOCONF
1092 def do_savedefconfig(self):
1093 """Update the .config and run 'make savedefconfig'."""
1095 (updated, suspicious, log) = self.parser.update_dotconfig()
1097 self.suspicious_boards.add(self.defconfig)
1100 if not self.options.force_sync and not updated:
1104 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
1105 "Syncing by savedefconfig...\n")
1107 self.log += "Syncing by savedefconfig (forced by option)...\n"
1109 cmd = list(self.make_cmd)
1110 cmd.append('savedefconfig')
1111 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1112 stderr=subprocess.PIPE)
1113 self.state = STATE_SAVEDEFCONFIG
1115 def update_defconfig(self):
1116 """Update the input defconfig and go back to the idle state."""
1118 log = self.parser.check_defconfig()
1120 self.suspicious_boards.add(self.defconfig)
1122 orig_defconfig = os.path.join('configs', self.defconfig)
1123 new_defconfig = os.path.join(self.build_dir, 'defconfig')
1124 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
1127 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
1128 "defconfig was updated.\n")
1130 if not self.options.dry_run and updated:
1131 shutil.move(new_defconfig, orig_defconfig)
1134 def finish(self, success):
1135 """Display log along with progress and go to the idle state.
1138 success: Should be True when the defconfig was processed
1139 successfully, or False when it fails.
1141 # output at least 30 characters to hide the "* defconfigs out of *".
1142 log = self.defconfig.ljust(30) + '\n'
1144 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
1145 # Some threads are running in parallel.
1146 # Print log atomically to not mix up logs from different threads.
1147 print >> (sys.stdout if success else sys.stderr), log
1150 if self.options.exit_on_error:
1151 sys.exit("Exit on error.")
1152 # If --exit-on-error flag is not set, skip this board and continue.
1153 # Record the failed board.
1154 self.failed_boards.add(self.defconfig)
1157 self.progress.show()
1158 self.state = STATE_IDLE
1160 def get_failed_boards(self):
1161 """Returns a set of failed boards (defconfigs) in this slot.
1163 return self.failed_boards
1165 def get_suspicious_boards(self):
1166 """Returns a set of boards (defconfigs) with possible misconversion.
1168 return self.suspicious_boards - self.failed_boards
1172 """Controller of the array of subprocess slots."""
1174 def __init__(self, configs, options, progress, reference_src_dir):
1175 """Create a new slots controller.
1178 configs: A list of CONFIGs to move.
1179 options: option flags.
1180 progress: A progress indicator.
1181 reference_src_dir: Determine the true starting config state from this
1184 self.options = options
1186 devnull = get_devnull()
1187 make_cmd = get_make_cmd()
1188 for i in range(options.jobs):
1189 self.slots.append(Slot(configs, options, progress, devnull,
1190 make_cmd, reference_src_dir))
1192 def add(self, defconfig):
1193 """Add a new subprocess if a vacant slot is found.
1196 defconfig: defconfig name to be put into.
1199 Return True on success or False on failure
1201 for slot in self.slots:
1202 if slot.add(defconfig):
1206 def available(self):
1207 """Check if there is a vacant slot.
1210 Return True if at lease one vacant slot is found, False otherwise.
1212 for slot in self.slots:
1218 """Check if all slots are vacant.
1221 Return True if all the slots are vacant, False otherwise.
1224 for slot in self.slots:
1229 def show_failed_boards(self):
1230 """Display all of the failed boards (defconfigs)."""
1232 output_file = 'moveconfig.failed'
1234 for slot in self.slots:
1235 boards |= slot.get_failed_boards()
1238 boards = '\n'.join(boards) + '\n'
1239 msg = "The following boards were not processed due to error:\n"
1241 msg += "(the list has been saved in %s)\n" % output_file
1242 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1245 with open(output_file, 'w') as f:
1248 def show_suspicious_boards(self):
1249 """Display all boards (defconfigs) with possible misconversion."""
1251 output_file = 'moveconfig.suspicious'
1253 for slot in self.slots:
1254 boards |= slot.get_suspicious_boards()
1257 boards = '\n'.join(boards) + '\n'
1258 msg = "The following boards might have been converted incorrectly.\n"
1259 msg += "It is highly recommended to check them manually:\n"
1261 msg += "(the list has been saved in %s)\n" % output_file
1262 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1265 with open(output_file, 'w') as f:
1268 class ReferenceSource:
1270 """Reference source against which original configs should be parsed."""
1272 def __init__(self, commit):
1273 """Create a reference source directory based on a specified commit.
1276 commit: commit to git-clone
1278 self.src_dir = tempfile.mkdtemp()
1279 print "Cloning git repo to a separate work directory..."
1280 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1282 print "Checkout '%s' to build the original autoconf.mk." % \
1283 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1284 subprocess.check_output(['git', 'checkout', commit],
1285 stderr=subprocess.STDOUT, cwd=self.src_dir)
1288 """Delete the reference source directory
1290 This function makes sure the temporary directory is cleaned away
1291 even if Python suddenly dies due to error. It should be done in here
1292 because it is guaranteed the destructor is always invoked when the
1293 instance of the class gets unreferenced.
1295 shutil.rmtree(self.src_dir)
1298 """Return the absolute path to the reference source directory."""
1302 def move_config(configs, options):
1303 """Move config options to defconfig files.
1306 configs: A list of CONFIGs to move.
1307 options: option flags
1309 if len(configs) == 0:
1310 if options.force_sync:
1311 print 'No CONFIG is specified. You are probably syncing defconfigs.',
1313 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1315 print 'Move ' + ', '.join(configs),
1316 print '(jobs: %d)\n' % options.jobs
1319 reference_src = ReferenceSource(options.git_ref)
1320 reference_src_dir = reference_src.get_dir()
1322 reference_src_dir = None
1324 if options.defconfigs:
1325 defconfigs = get_matched_defconfigs(options.defconfigs)
1327 defconfigs = get_all_defconfigs()
1329 progress = Progress(len(defconfigs))
1330 slots = Slots(configs, options, progress, reference_src_dir)
1332 # Main loop to process defconfig files:
1333 # Add a new subprocess into a vacant slot.
1334 # Sleep if there is no available slot.
1335 for defconfig in defconfigs:
1336 while not slots.add(defconfig):
1337 while not slots.available():
1338 # No available slot: sleep for a while
1339 time.sleep(SLEEP_TIME)
1341 # wait until all the subprocesses finish
1342 while not slots.empty():
1343 time.sleep(SLEEP_TIME)
1346 slots.show_failed_boards()
1347 slots.show_suspicious_boards()
1351 cpu_count = multiprocessing.cpu_count()
1352 except NotImplementedError:
1355 parser = optparse.OptionParser()
1357 parser.add_option('-c', '--color', action='store_true', default=False,
1358 help='display the log in color')
1359 parser.add_option('-C', '--commit', action='store_true', default=False,
1360 help='Create a git commit for the operation')
1361 parser.add_option('-d', '--defconfigs', type='string',
1362 help='a file containing a list of defconfigs to move, '
1363 "one per line (for example 'snow_defconfig') "
1364 "or '-' to read from stdin")
1365 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1366 help='perform a trial run (show log with no changes)')
1367 parser.add_option('-e', '--exit-on-error', action='store_true',
1369 help='exit immediately on any error')
1370 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1371 help='force sync by savedefconfig')
1372 parser.add_option('-S', '--spl', action='store_true', default=False,
1373 help='parse config options defined for SPL build')
1374 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1375 action='store_true', default=False,
1376 help='only cleanup the headers')
1377 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1378 help='the number of jobs to run simultaneously')
1379 parser.add_option('-r', '--git-ref', type='string',
1380 help='the git ref to clone for building the autoconf.mk')
1381 parser.add_option('-y', '--yes', action='store_true', default=False,
1382 help="respond 'yes' to any prompts")
1383 parser.add_option('-v', '--verbose', action='store_true', default=False,
1384 help='show any build errors as boards are built')
1385 parser.usage += ' CONFIG ...'
1387 (options, configs) = parser.parse_args()
1389 if len(configs) == 0 and not options.force_sync:
1390 parser.print_usage()
1393 # prefix the option name with CONFIG_ if missing
1394 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1395 for config in configs ]
1397 check_top_directory()
1399 if not options.cleanup_headers_only:
1400 check_clean_directory()
1401 update_cross_compile(options.color)
1402 move_config(configs, options)
1405 cleanup_headers(configs, options)
1406 cleanup_extra_options(configs, options)
1407 cleanup_whitelist(configs, options)
1408 cleanup_readme(configs, options)
1411 subprocess.call(['git', 'add', '-u'])
1413 msg = 'Convert %s %sto Kconfig' % (configs[0],
1414 'et al ' if len(configs) > 1 else '')
1415 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' %
1416 '\n '.join(configs))
1418 msg = 'configs: Resync with savedefconfig'
1419 msg += '\n\nRsync all defconfig files using moveconfig.py'
1420 subprocess.call(['git', 'commit', '-s', '-m', msg])
1422 if __name__ == '__main__':