3 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5 # SPDX-License-Identifier: GPL-2.0+
9 Move config options from headers to defconfig files.
11 Since Kconfig was introduced to U-Boot, we have worked on moving
12 config options from headers to Kconfig (defconfig).
14 This tool intends to help this tremendous work.
20 First, you must edit the Kconfig to add the menu entries for the configs
23 And then run this tool giving CONFIG names you want to move.
24 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25 simply type as follows:
27 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
29 The tool walks through all the defconfig files and move the given CONFIGs.
31 The log is also displayed on the terminal.
33 The log is printed for each defconfig as follows:
41 <defconfig_name> is the name of the defconfig.
43 <action*> shows what the tool did for that defconfig.
44 It looks like one of the following:
47 This config option was moved to the defconfig
49 - CONFIG_... is not defined in Kconfig. Do nothing.
50 The entry for this CONFIG was not found in Kconfig. The option is not
51 defined in the config header, either. So, this case can be just skipped.
53 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
54 This option is defined in the config header, but its entry was not found
56 There are two common cases:
57 - You forgot to create an entry for the CONFIG before running
58 this tool, or made a typo in a CONFIG passed to this tool.
59 - The entry was hidden due to unmet 'depends on'.
60 The tool does not know if the result is reasonable, so please check it
63 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
64 The define in the config header matched the one in Kconfig.
65 We do not need to touch it.
67 - Compiler is missing. Do nothing.
68 The compiler specified for this architecture was not found
69 in your PATH environment.
70 (If -e option is passed, the tool exits immediately.)
73 An error occurred during processing this defconfig. Skipped.
74 (If -e option is passed, the tool exits immediately on error.)
76 Finally, you will be asked, Clean up headers? [y/n]:
78 If you say 'y' here, the unnecessary config defines are removed
79 from the config headers (include/configs/*.h).
80 It just uses the regex method, so you should not rely on it.
81 Just in case, please do 'git diff' to see what happened.
87 This tool runs configuration and builds include/autoconf.mk for every
88 defconfig. The config options defined in Kconfig appear in the .config
89 file (unless they are hidden because of unmet dependency.)
90 On the other hand, the config options defined by board headers are seen
91 in include/autoconf.mk. The tool looks for the specified options in both
92 of them to decide the appropriate action for the options. If the given
93 config option is found in the .config, but its value does not match the
94 one from the board header, the config option in the .config is replaced
95 with the define in the board header. Then, the .config is synced by
96 "make savedefconfig" and the defconfig is updated with it.
98 For faster processing, this tool handles multi-threading. It creates
99 separate build directories where the out-of-tree build is run. The
100 temporary build directories are automatically created and deleted as
101 needed. The number of threads are chosen based on the number of the CPU
102 cores of your system although you can change it via -j (--jobs) option.
108 Appropriate toolchain are necessary to generate include/autoconf.mk
109 for all the architectures supported by U-Boot. Most of them are available
110 at the kernel.org site, some are not provided by kernel.org.
112 The default per-arch CROSS_COMPILE used by this tool is specified by
113 the list below, CROSS_COMPILE. You may wish to update the list to
114 use your own. Instead of modifying the list directly, you can give
115 them via environments.
122 Surround each portion of the log with escape sequences to display it
123 in color on the terminal.
126 Create a git commit with the changes when the operation is complete. A
127 standard commit message is used which may need to be edited.
130 Specify a file containing a list of defconfigs to move. The defconfig
131 files can be given with shell-style wildcards.
134 Perform a trial run that does not make any changes. It is useful to
135 see what is going to happen before one actually runs it.
138 Exit immediately if Make exits with a non-zero status while processing
142 Do "make savedefconfig" forcibly for all the defconfig files.
143 If not specified, "make savedefconfig" only occurs for cases
144 where at least one CONFIG was moved.
147 Look for moved config options in spl/include/autoconf.mk instead of
148 include/autoconf.mk. This is useful for moving options for SPL build
149 because SPL related options (mostly prefixed with CONFIG_SPL_) are
150 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
153 Only cleanup the headers; skip the defconfig processing
156 Specify the number of threads to run simultaneously. If not specified,
157 the number of threads is the same as the number of CPU cores.
160 Specify the git ref to clone for building the autoconf.mk. If unspecified
161 use the CWD. This is useful for when changes to the Kconfig affect the
162 default values and you want to capture the state of the defconfig from
163 before that change was in effect. If in doubt, specify a ref pre-Kconfig
164 changes (use HEAD if Kconfig changes are not committed). Worst case it will
165 take a bit longer to run, but will always do the right thing.
168 Show any build errors as boards are built
171 Instead of prompting, automatically go ahead with all operations. This
172 includes cleaning up headers and CONFIG_SYS_EXTRA_OPTIONS.
174 To see the complete list of supported options, run
176 $ tools/moveconfig.py -h
185 import multiprocessing
195 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
198 # Here is the list of cross-tools I use.
199 # Most of them are available at kernel.org
200 # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
201 # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
202 # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
203 # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
204 # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
207 'aarch64': 'aarch64-linux-',
208 'arm': 'arm-unknown-linux-gnueabi-',
209 'm68k': 'm68k-linux-',
210 'microblaze': 'microblaze-linux-',
211 'mips': 'mips-linux-',
212 'nds32': 'nds32le-linux-',
213 'nios2': 'nios2-linux-gnu-',
214 'powerpc': 'powerpc-linux-',
215 'sh': 'sh-linux-gnu-',
216 'x86': 'i386-linux-',
217 'xtensa': 'xtensa-linux-'
223 STATE_SAVEDEFCONFIG = 3
227 ACTION_NO_ENTRY_WARN = 2
235 COLOR_PURPLE = '0;35'
237 COLOR_LIGHT_GRAY = '0;37'
238 COLOR_DARK_GRAY = '1;30'
239 COLOR_LIGHT_RED = '1;31'
240 COLOR_LIGHT_GREEN = '1;32'
241 COLOR_YELLOW = '1;33'
242 COLOR_LIGHT_BLUE = '1;34'
243 COLOR_LIGHT_PURPLE = '1;35'
244 COLOR_LIGHT_CYAN = '1;36'
247 ### helper functions ###
249 """Get the file object of '/dev/null' device."""
251 devnull = subprocess.DEVNULL # py3k
252 except AttributeError:
253 devnull = open(os.devnull, 'wb')
256 def check_top_directory():
257 """Exit if we are not at the top of source directory."""
258 for f in ('README', 'Licenses'):
259 if not os.path.exists(f):
260 sys.exit('Please run at the top of source directory.')
262 def check_clean_directory():
263 """Exit if the source tree is not clean."""
264 for f in ('.config', 'include/config'):
265 if os.path.exists(f):
266 sys.exit("source tree is not clean, please run 'make mrproper'")
269 """Get the command name of GNU Make.
271 U-Boot needs GNU Make for building, but the command name is not
272 necessarily "make". (for example, "gmake" on FreeBSD).
273 Returns the most appropriate command name on your system.
275 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
276 ret = process.communicate()
277 if process.returncode:
278 sys.exit('GNU Make not found')
279 return ret[0].rstrip()
281 def get_matched_defconfigs(defconfigs_file):
282 """Get all the defconfig files that match the patterns in a file."""
284 for i, line in enumerate(open(defconfigs_file)):
287 continue # skip blank lines silently
288 pattern = os.path.join('configs', line)
289 matched = glob.glob(pattern) + glob.glob(pattern + '_defconfig')
291 print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
292 (defconfigs_file, i + 1, line)
294 defconfigs += matched
296 # use set() to drop multiple matching
297 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ]
299 def get_all_defconfigs():
300 """Get all the defconfig files under the configs/ directory."""
302 for (dirpath, dirnames, filenames) in os.walk('configs'):
303 dirpath = dirpath[len('configs') + 1:]
304 for filename in fnmatch.filter(filenames, '*_defconfig'):
305 defconfigs.append(os.path.join(dirpath, filename))
309 def color_text(color_enabled, color, string):
310 """Return colored string."""
312 # LF should not be surrounded by the escape sequence.
313 # Otherwise, additional whitespace or line-feed might be printed.
314 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
315 for s in string.split('\n') ])
319 def show_diff(a, b, file_path, color_enabled):
320 """Show unidified diff.
323 a: A list of lines (before)
324 b: A list of lines (after)
325 file_path: Path to the file
326 color_enabled: Display the diff in color
329 diff = difflib.unified_diff(a, b,
330 fromfile=os.path.join('a', file_path),
331 tofile=os.path.join('b', file_path))
334 if line[0] == '-' and line[1] != '-':
335 print color_text(color_enabled, COLOR_RED, line),
336 elif line[0] == '+' and line[1] != '+':
337 print color_text(color_enabled, COLOR_GREEN, line),
341 def update_cross_compile(color_enabled):
342 """Update per-arch CROSS_COMPILE via environment variables
344 The default CROSS_COMPILE values are available
345 in the CROSS_COMPILE list above.
347 You can override them via environment variables
348 CROSS_COMPILE_{ARCH}.
350 For example, if you want to override toolchain prefixes
351 for ARM and PowerPC, you can do as follows in your shell:
353 export CROSS_COMPILE_ARM=...
354 export CROSS_COMPILE_POWERPC=...
356 Then, this function checks if specified compilers really exist in your
361 for arch in os.listdir('arch'):
362 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
365 # arm64 is a special case
366 archs.append('aarch64')
369 env = 'CROSS_COMPILE_' + arch.upper()
370 cross_compile = os.environ.get(env)
371 if not cross_compile:
372 cross_compile = CROSS_COMPILE.get(arch, '')
374 for path in os.environ["PATH"].split(os.pathsep):
375 gcc_path = os.path.join(path, cross_compile + 'gcc')
376 if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
379 print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
380 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
381 % (cross_compile, arch))
384 CROSS_COMPILE[arch] = cross_compile
386 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
388 """Extend matched lines if desired patterns are found before/after already
392 lines: A list of lines handled.
393 matched: A list of line numbers that have been already matched.
394 (will be updated by this function)
395 pre_patterns: A list of regular expression that should be matched as
397 post_patterns: A list of regular expression that should be matched as
399 extend_pre: Add the line number of matched preamble to the matched list.
400 extend_post: Add the line number of matched postamble to the matched list.
402 extended_matched = []
415 for p in pre_patterns:
416 if p.search(lines[i - 1]):
422 for p in post_patterns:
423 if p.search(lines[j]):
430 extended_matched.append(i - 1)
432 extended_matched.append(j)
434 matched += extended_matched
437 def confirm(options, prompt):
440 choice = raw_input('{} [y/n]: '.format(prompt))
441 choice = choice.lower()
443 if choice == 'y' or choice == 'n':
451 def cleanup_one_header(header_path, patterns, options):
452 """Clean regex-matched lines away from a file.
455 header_path: path to the cleaned file.
456 patterns: list of regex patterns. Any lines matching to these
457 patterns are deleted.
458 options: option flags.
460 with open(header_path) as f:
461 lines = f.readlines()
464 for i, line in enumerate(lines):
465 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
468 for pattern in patterns:
469 if pattern.search(line):
476 # remove empty #ifdef ... #endif, successive blank lines
477 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
478 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
479 pattern_endif = re.compile(r'#\s*endif\W') # #endif
480 pattern_blank = re.compile(r'^\s*$') # empty line
483 old_matched = copy.copy(matched)
484 extend_matched_lines(lines, matched, [pattern_if],
485 [pattern_endif], True, True)
486 extend_matched_lines(lines, matched, [pattern_elif],
487 [pattern_elif, pattern_endif], True, False)
488 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
489 [pattern_blank], False, True)
490 extend_matched_lines(lines, matched, [pattern_blank],
491 [pattern_elif, pattern_endif], True, False)
492 extend_matched_lines(lines, matched, [pattern_blank],
493 [pattern_blank], True, False)
494 if matched == old_matched:
497 tolines = copy.copy(lines)
499 for i in reversed(matched):
502 show_diff(lines, tolines, header_path, options.color)
507 with open(header_path, 'w') as f:
511 def cleanup_headers(configs, options):
512 """Delete config defines from board headers.
515 configs: A list of CONFIGs to remove.
516 options: option flags.
518 if not confirm(options, 'Clean up headers?'):
522 for config in configs:
523 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
524 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
526 for dir in 'include', 'arch', 'board':
527 for (dirpath, dirnames, filenames) in os.walk(dir):
528 if dirpath == os.path.join('include', 'generated'):
530 for filename in filenames:
531 if not fnmatch.fnmatch(filename, '*~'):
532 cleanup_one_header(os.path.join(dirpath, filename),
535 def cleanup_one_extra_option(defconfig_path, configs, options):
536 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
539 defconfig_path: path to the cleaned defconfig file.
540 configs: A list of CONFIGs to remove.
541 options: option flags.
544 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
547 with open(defconfig_path) as f:
548 lines = f.readlines()
550 for i, line in enumerate(lines):
551 if line.startswith(start) and line.endswith(end):
554 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
557 old_tokens = line[len(start):-len(end)].split(',')
560 for token in old_tokens:
561 pos = token.find('=')
562 if not (token[:pos] if pos >= 0 else token) in configs:
563 new_tokens.append(token)
565 if new_tokens == old_tokens:
568 tolines = copy.copy(lines)
571 tolines[i] = start + ','.join(new_tokens) + end
575 show_diff(lines, tolines, defconfig_path, options.color)
580 with open(defconfig_path, 'w') as f:
584 def cleanup_extra_options(configs, options):
585 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
588 configs: A list of CONFIGs to remove.
589 options: option flags.
591 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
594 configs = [ config[len('CONFIG_'):] for config in configs ]
596 defconfigs = get_all_defconfigs()
598 for defconfig in defconfigs:
599 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
602 def cleanup_whitelist(configs, options):
603 """Delete config whitelist entries
606 configs: A list of CONFIGs to remove.
607 options: option flags.
609 if not confirm(options, 'Clean up whitelist entries?'):
612 with open(os.path.join('scripts', 'config_whitelist.txt')) as f:
613 lines = f.readlines()
615 lines = [x for x in lines if x.strip() not in configs]
617 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f:
618 f.write(''.join(lines))
620 def find_matching(patterns, line):
626 def cleanup_readme(configs, options):
627 """Delete config description in README
630 configs: A list of CONFIGs to remove.
631 options: option flags.
633 if not confirm(options, 'Clean up README?'):
637 for config in configs:
638 patterns.append(re.compile(r'^\s+%s' % config))
640 with open('README') as f:
641 lines = f.readlines()
647 found = find_matching(patterns, line)
651 if found and re.search(r'^\s+CONFIG', line):
655 newlines.append(line)
657 with open('README', 'w') as f:
658 f.write(''.join(newlines))
664 """Progress Indicator"""
666 def __init__(self, total):
667 """Create a new progress indicator.
670 total: A number of defconfig files to process.
676 """Increment the number of processed defconfig files."""
681 """Display the progress."""
682 print ' %d defconfigs out of %d\r' % (self.current, self.total),
687 """A parser of .config and include/autoconf.mk."""
689 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
690 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
692 def __init__(self, configs, options, build_dir):
693 """Create a new parser.
696 configs: A list of CONFIGs to move.
697 options: option flags.
698 build_dir: Build directory.
700 self.configs = configs
701 self.options = options
702 self.dotconfig = os.path.join(build_dir, '.config')
703 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
704 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
706 self.config_autoconf = os.path.join(build_dir, 'include', 'config',
708 self.defconfig = os.path.join(build_dir, 'defconfig')
710 def get_cross_compile(self):
711 """Parse .config file and return CROSS_COMPILE.
714 A string storing the compiler prefix for the architecture.
715 Return a NULL string for architectures that do not require
716 compiler prefix (Sandbox and native build is the case).
717 Return None if the specified compiler is missing in your PATH.
718 Caller should distinguish '' and None.
722 for line in open(self.dotconfig):
723 m = self.re_arch.match(line)
727 m = self.re_cpu.match(line)
735 if arch == 'arm' and cpu == 'armv8':
738 return CROSS_COMPILE.get(arch, None)
740 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
741 """Parse .config, defconfig, include/autoconf.mk for one config.
743 This function looks for the config options in the lines from
744 defconfig, .config, and include/autoconf.mk in order to decide
745 which action should be taken for this defconfig.
748 config: CONFIG name to parse.
749 dotconfig_lines: lines from the .config file.
750 autoconf_lines: lines from the include/autoconf.mk file.
753 A tupple of the action for this defconfig and the line
754 matched for the config.
756 not_set = '# %s is not set' % config
758 for line in autoconf_lines:
760 if line.startswith(config + '='):
766 for line in dotconfig_lines:
768 if line.startswith(config + '=') or line == not_set:
772 if new_val == not_set:
773 return (ACTION_NO_ENTRY, config)
775 return (ACTION_NO_ENTRY_WARN, config)
777 # If this CONFIG is neither bool nor trisate
778 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
779 # tools/scripts/define2mk.sed changes '1' to 'y'.
780 # This is a problem if the CONFIG is int type.
781 # Check the type in Kconfig and handle it correctly.
782 if new_val[-2:] == '=y':
783 new_val = new_val[:-1] + '1'
785 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
788 def update_dotconfig(self):
789 """Parse files for the config options and update the .config.
791 This function parses the generated .config and include/autoconf.mk
792 searching the target options.
793 Move the config option(s) to the .config as needed.
796 defconfig: defconfig name.
799 Return a tuple of (updated flag, log string).
800 The "updated flag" is True if the .config was updated, False
801 otherwise. The "log string" shows what happend to the .config.
807 rm_files = [self.config_autoconf, self.autoconf]
810 if os.path.exists(self.spl_autoconf):
811 autoconf_path = self.spl_autoconf
812 rm_files.append(self.spl_autoconf)
816 return (updated, suspicious,
817 color_text(self.options.color, COLOR_BROWN,
818 "SPL is not enabled. Skipped.") + '\n')
820 autoconf_path = self.autoconf
822 with open(self.dotconfig) as f:
823 dotconfig_lines = f.readlines()
825 with open(autoconf_path) as f:
826 autoconf_lines = f.readlines()
828 for config in self.configs:
829 result = self.parse_one_config(config, dotconfig_lines,
831 results.append(result)
835 for (action, value) in results:
836 if action == ACTION_MOVE:
837 actlog = "Move '%s'" % value
838 log_color = COLOR_LIGHT_GREEN
839 elif action == ACTION_NO_ENTRY:
840 actlog = "%s is not defined in Kconfig. Do nothing." % value
841 log_color = COLOR_LIGHT_BLUE
842 elif action == ACTION_NO_ENTRY_WARN:
843 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
844 log_color = COLOR_YELLOW
846 elif action == ACTION_NO_CHANGE:
847 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
849 log_color = COLOR_LIGHT_PURPLE
850 elif action == ACTION_SPL_NOT_EXIST:
851 actlog = "SPL is not enabled for this defconfig. Skip."
852 log_color = COLOR_PURPLE
854 sys.exit("Internal Error. This should not happen.")
856 log += color_text(self.options.color, log_color, actlog) + '\n'
858 with open(self.dotconfig, 'a') as f:
859 for (action, value) in results:
860 if action == ACTION_MOVE:
861 f.write(value + '\n')
864 self.results = results
868 return (updated, suspicious, log)
870 def check_defconfig(self):
871 """Check the defconfig after savedefconfig
874 Return additional log if moved CONFIGs were removed again by
875 'make savedefconfig'.
880 with open(self.defconfig) as f:
881 defconfig_lines = f.readlines()
883 for (action, value) in self.results:
884 if action != ACTION_MOVE:
886 if not value + '\n' in defconfig_lines:
887 log += color_text(self.options.color, COLOR_YELLOW,
888 "'%s' was removed by savedefconfig.\n" %
895 """A slot to store a subprocess.
897 Each instance of this class handles one subprocess.
898 This class is useful to control multiple threads
899 for faster processing.
902 def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir):
903 """Create a new process slot.
906 configs: A list of CONFIGs to move.
907 options: option flags.
908 progress: A progress indicator.
909 devnull: A file object of '/dev/null'.
910 make_cmd: command name of GNU Make.
911 reference_src_dir: Determine the true starting config state from this
914 self.options = options
915 self.progress = progress
916 self.build_dir = tempfile.mkdtemp()
917 self.devnull = devnull
918 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
919 self.reference_src_dir = reference_src_dir
920 self.parser = KconfigParser(configs, options, self.build_dir)
921 self.state = STATE_IDLE
922 self.failed_boards = set()
923 self.suspicious_boards = set()
926 """Delete the working directory
928 This function makes sure the temporary directory is cleaned away
929 even if Python suddenly dies due to error. It should be done in here
930 because it is guaranteed the destructor is always invoked when the
931 instance of the class gets unreferenced.
933 If the subprocess is still running, wait until it finishes.
935 if self.state != STATE_IDLE:
936 while self.ps.poll() == None:
938 shutil.rmtree(self.build_dir)
940 def add(self, defconfig):
941 """Assign a new subprocess for defconfig and add it to the slot.
943 If the slot is vacant, create a new subprocess for processing the
944 given defconfig and add it to the slot. Just returns False if
945 the slot is occupied (i.e. the current subprocess is still running).
948 defconfig: defconfig name.
951 Return True on success or False on failure
953 if self.state != STATE_IDLE:
956 self.defconfig = defconfig
958 self.current_src_dir = self.reference_src_dir
963 """Check the status of the subprocess and handle it as needed.
965 Returns True if the slot is vacant (i.e. in idle state).
966 If the configuration is successfully finished, assign a new
967 subprocess to build include/autoconf.mk.
968 If include/autoconf.mk is generated, invoke the parser to
969 parse the .config and the include/autoconf.mk, moving
970 config options to the .config as needed.
971 If the .config was updated, run "make savedefconfig" to sync
972 it, update the original defconfig, and then set the slot back
976 Return True if the subprocess is terminated, False otherwise
978 if self.state == STATE_IDLE:
981 if self.ps.poll() == None:
984 if self.ps.poll() != 0:
986 elif self.state == STATE_DEFCONFIG:
987 if self.reference_src_dir and not self.current_src_dir:
988 self.do_savedefconfig()
991 elif self.state == STATE_AUTOCONF:
992 if self.current_src_dir:
993 self.current_src_dir = None
996 self.do_savedefconfig()
997 elif self.state == STATE_SAVEDEFCONFIG:
998 self.update_defconfig()
1000 sys.exit("Internal Error. This should not happen.")
1002 return True if self.state == STATE_IDLE else False
1004 def handle_error(self):
1005 """Handle error cases."""
1007 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
1008 "Failed to process.\n")
1009 if self.options.verbose:
1010 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
1011 self.ps.stderr.read())
1014 def do_defconfig(self):
1015 """Run 'make <board>_defconfig' to create the .config file."""
1017 cmd = list(self.make_cmd)
1018 cmd.append(self.defconfig)
1019 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1020 stderr=subprocess.PIPE,
1021 cwd=self.current_src_dir)
1022 self.state = STATE_DEFCONFIG
1024 def do_autoconf(self):
1025 """Run 'make include/config/auto.conf'."""
1027 self.cross_compile = self.parser.get_cross_compile()
1028 if self.cross_compile is None:
1029 self.log += color_text(self.options.color, COLOR_YELLOW,
1030 "Compiler is missing. Do nothing.\n")
1034 cmd = list(self.make_cmd)
1035 if self.cross_compile:
1036 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
1037 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
1038 cmd.append('include/config/auto.conf')
1039 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1040 stderr=subprocess.PIPE,
1041 cwd=self.current_src_dir)
1042 self.state = STATE_AUTOCONF
1044 def do_savedefconfig(self):
1045 """Update the .config and run 'make savedefconfig'."""
1047 (updated, suspicious, log) = self.parser.update_dotconfig()
1049 self.suspicious_boards.add(self.defconfig)
1052 if not self.options.force_sync and not updated:
1056 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
1057 "Syncing by savedefconfig...\n")
1059 self.log += "Syncing by savedefconfig (forced by option)...\n"
1061 cmd = list(self.make_cmd)
1062 cmd.append('savedefconfig')
1063 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1064 stderr=subprocess.PIPE)
1065 self.state = STATE_SAVEDEFCONFIG
1067 def update_defconfig(self):
1068 """Update the input defconfig and go back to the idle state."""
1070 log = self.parser.check_defconfig()
1072 self.suspicious_boards.add(self.defconfig)
1074 orig_defconfig = os.path.join('configs', self.defconfig)
1075 new_defconfig = os.path.join(self.build_dir, 'defconfig')
1076 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
1079 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
1080 "defconfig was updated.\n")
1082 if not self.options.dry_run and updated:
1083 shutil.move(new_defconfig, orig_defconfig)
1086 def finish(self, success):
1087 """Display log along with progress and go to the idle state.
1090 success: Should be True when the defconfig was processed
1091 successfully, or False when it fails.
1093 # output at least 30 characters to hide the "* defconfigs out of *".
1094 log = self.defconfig.ljust(30) + '\n'
1096 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
1097 # Some threads are running in parallel.
1098 # Print log atomically to not mix up logs from different threads.
1099 print >> (sys.stdout if success else sys.stderr), log
1102 if self.options.exit_on_error:
1103 sys.exit("Exit on error.")
1104 # If --exit-on-error flag is not set, skip this board and continue.
1105 # Record the failed board.
1106 self.failed_boards.add(self.defconfig)
1109 self.progress.show()
1110 self.state = STATE_IDLE
1112 def get_failed_boards(self):
1113 """Returns a set of failed boards (defconfigs) in this slot.
1115 return self.failed_boards
1117 def get_suspicious_boards(self):
1118 """Returns a set of boards (defconfigs) with possible misconversion.
1120 return self.suspicious_boards - self.failed_boards
1124 """Controller of the array of subprocess slots."""
1126 def __init__(self, configs, options, progress, reference_src_dir):
1127 """Create a new slots controller.
1130 configs: A list of CONFIGs to move.
1131 options: option flags.
1132 progress: A progress indicator.
1133 reference_src_dir: Determine the true starting config state from this
1136 self.options = options
1138 devnull = get_devnull()
1139 make_cmd = get_make_cmd()
1140 for i in range(options.jobs):
1141 self.slots.append(Slot(configs, options, progress, devnull,
1142 make_cmd, reference_src_dir))
1144 def add(self, defconfig):
1145 """Add a new subprocess if a vacant slot is found.
1148 defconfig: defconfig name to be put into.
1151 Return True on success or False on failure
1153 for slot in self.slots:
1154 if slot.add(defconfig):
1158 def available(self):
1159 """Check if there is a vacant slot.
1162 Return True if at lease one vacant slot is found, False otherwise.
1164 for slot in self.slots:
1170 """Check if all slots are vacant.
1173 Return True if all the slots are vacant, False otherwise.
1176 for slot in self.slots:
1181 def show_failed_boards(self):
1182 """Display all of the failed boards (defconfigs)."""
1184 output_file = 'moveconfig.failed'
1186 for slot in self.slots:
1187 boards |= slot.get_failed_boards()
1190 boards = '\n'.join(boards) + '\n'
1191 msg = "The following boards were not processed due to error:\n"
1193 msg += "(the list has been saved in %s)\n" % output_file
1194 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1197 with open(output_file, 'w') as f:
1200 def show_suspicious_boards(self):
1201 """Display all boards (defconfigs) with possible misconversion."""
1203 output_file = 'moveconfig.suspicious'
1205 for slot in self.slots:
1206 boards |= slot.get_suspicious_boards()
1209 boards = '\n'.join(boards) + '\n'
1210 msg = "The following boards might have been converted incorrectly.\n"
1211 msg += "It is highly recommended to check them manually:\n"
1213 msg += "(the list has been saved in %s)\n" % output_file
1214 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1217 with open(output_file, 'w') as f:
1220 class ReferenceSource:
1222 """Reference source against which original configs should be parsed."""
1224 def __init__(self, commit):
1225 """Create a reference source directory based on a specified commit.
1228 commit: commit to git-clone
1230 self.src_dir = tempfile.mkdtemp()
1231 print "Cloning git repo to a separate work directory..."
1232 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1234 print "Checkout '%s' to build the original autoconf.mk." % \
1235 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1236 subprocess.check_output(['git', 'checkout', commit],
1237 stderr=subprocess.STDOUT, cwd=self.src_dir)
1240 """Delete the reference source directory
1242 This function makes sure the temporary directory is cleaned away
1243 even if Python suddenly dies due to error. It should be done in here
1244 because it is guaranteed the destructor is always invoked when the
1245 instance of the class gets unreferenced.
1247 shutil.rmtree(self.src_dir)
1250 """Return the absolute path to the reference source directory."""
1254 def move_config(configs, options):
1255 """Move config options to defconfig files.
1258 configs: A list of CONFIGs to move.
1259 options: option flags
1261 if len(configs) == 0:
1262 if options.force_sync:
1263 print 'No CONFIG is specified. You are probably syncing defconfigs.',
1265 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1267 print 'Move ' + ', '.join(configs),
1268 print '(jobs: %d)\n' % options.jobs
1271 reference_src = ReferenceSource(options.git_ref)
1272 reference_src_dir = reference_src.get_dir()
1274 reference_src_dir = None
1276 if options.defconfigs:
1277 defconfigs = get_matched_defconfigs(options.defconfigs)
1279 defconfigs = get_all_defconfigs()
1281 progress = Progress(len(defconfigs))
1282 slots = Slots(configs, options, progress, reference_src_dir)
1284 # Main loop to process defconfig files:
1285 # Add a new subprocess into a vacant slot.
1286 # Sleep if there is no available slot.
1287 for defconfig in defconfigs:
1288 while not slots.add(defconfig):
1289 while not slots.available():
1290 # No available slot: sleep for a while
1291 time.sleep(SLEEP_TIME)
1293 # wait until all the subprocesses finish
1294 while not slots.empty():
1295 time.sleep(SLEEP_TIME)
1298 slots.show_failed_boards()
1299 slots.show_suspicious_boards()
1303 cpu_count = multiprocessing.cpu_count()
1304 except NotImplementedError:
1307 parser = optparse.OptionParser()
1309 parser.add_option('-c', '--color', action='store_true', default=False,
1310 help='display the log in color')
1311 parser.add_option('-C', '--commit', action='store_true', default=False,
1312 help='Create a git commit for the operation')
1313 parser.add_option('-d', '--defconfigs', type='string',
1314 help='a file containing a list of defconfigs to move')
1315 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1316 help='perform a trial run (show log with no changes)')
1317 parser.add_option('-e', '--exit-on-error', action='store_true',
1319 help='exit immediately on any error')
1320 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1321 help='force sync by savedefconfig')
1322 parser.add_option('-S', '--spl', action='store_true', default=False,
1323 help='parse config options defined for SPL build')
1324 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1325 action='store_true', default=False,
1326 help='only cleanup the headers')
1327 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1328 help='the number of jobs to run simultaneously')
1329 parser.add_option('-r', '--git-ref', type='string',
1330 help='the git ref to clone for building the autoconf.mk')
1331 parser.add_option('-y', '--yes', action='store_true', default=False,
1332 help="respond 'yes' to any prompts")
1333 parser.add_option('-v', '--verbose', action='store_true', default=False,
1334 help='show any build errors as boards are built')
1335 parser.usage += ' CONFIG ...'
1337 (options, configs) = parser.parse_args()
1339 if len(configs) == 0 and not options.force_sync:
1340 parser.print_usage()
1343 # prefix the option name with CONFIG_ if missing
1344 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1345 for config in configs ]
1347 check_top_directory()
1349 if not options.cleanup_headers_only:
1350 check_clean_directory()
1351 update_cross_compile(options.color)
1352 move_config(configs, options)
1355 cleanup_headers(configs, options)
1356 cleanup_extra_options(configs, options)
1357 cleanup_whitelist(configs, options)
1358 cleanup_readme(configs, options)
1361 subprocess.call(['git', 'add', '-u'])
1363 msg = 'Convert %s %sto Kconfig' % (configs[0],
1364 'et al ' if len(configs) > 1 else '')
1365 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' %
1366 '\n '.join(configs))
1368 msg = 'configs: Resync with savedefconfig'
1369 msg += '\n\nRsync all defconfig files using moveconfig.py'
1370 subprocess.call(['git', 'commit', '-s', '-m', msg])
1372 if __name__ == '__main__':