3 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5 # SPDX-License-Identifier: GPL-2.0+
9 Move config options from headers to defconfig files.
11 Since Kconfig was introduced to U-Boot, we have worked on moving
12 config options from headers to Kconfig (defconfig).
14 This tool intends to help this tremendous work.
20 First, you must edit the Kconfig to add the menu entries for the configs
23 And then run this tool giving CONFIG names you want to move.
24 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25 simply type as follows:
27 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
29 The tool walks through all the defconfig files and move the given CONFIGs.
31 The log is also displayed on the terminal.
33 The log is printed for each defconfig as follows:
41 <defconfig_name> is the name of the defconfig.
43 <action*> shows what the tool did for that defconfig.
44 It looks like one of the following:
47 This config option was moved to the defconfig
49 - CONFIG_... is not defined in Kconfig. Do nothing.
50 The entry for this CONFIG was not found in Kconfig.
51 There are two common cases:
52 - You forgot to create an entry for the CONFIG before running
53 this tool, or made a typo in a CONFIG passed to this tool.
54 - The entry was hidden due to unmet 'depends on'.
55 This is correct behavior.
57 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
58 The define in the config header matched the one in Kconfig.
59 We do not need to touch it.
61 - Compiler is missing. Do nothing.
62 The compiler specified for this architecture was not found
63 in your PATH environment.
64 (If -e option is passed, the tool exits immediately.)
67 An error occurred during processing this defconfig. Skipped.
68 (If -e option is passed, the tool exits immediately on error.)
70 Finally, you will be asked, Clean up headers? [y/n]:
72 If you say 'y' here, the unnecessary config defines are removed
73 from the config headers (include/configs/*.h).
74 It just uses the regex method, so you should not rely on it.
75 Just in case, please do 'git diff' to see what happened.
81 This tool runs configuration and builds include/autoconf.mk for every
82 defconfig. The config options defined in Kconfig appear in the .config
83 file (unless they are hidden because of unmet dependency.)
84 On the other hand, the config options defined by board headers are seen
85 in include/autoconf.mk. The tool looks for the specified options in both
86 of them to decide the appropriate action for the options. If the given
87 config option is found in the .config, but its value does not match the
88 one from the board header, the config option in the .config is replaced
89 with the define in the board header. Then, the .config is synced by
90 "make savedefconfig" and the defconfig is updated with it.
92 For faster processing, this tool handles multi-threading. It creates
93 separate build directories where the out-of-tree build is run. The
94 temporary build directories are automatically created and deleted as
95 needed. The number of threads are chosen based on the number of the CPU
96 cores of your system although you can change it via -j (--jobs) option.
102 Appropriate toolchain are necessary to generate include/autoconf.mk
103 for all the architectures supported by U-Boot. Most of them are available
104 at the kernel.org site, some are not provided by kernel.org.
106 The default per-arch CROSS_COMPILE used by this tool is specified by
107 the list below, CROSS_COMPILE. You may wish to update the list to
108 use your own. Instead of modifying the list directly, you can give
109 them via environments.
116 Surround each portion of the log with escape sequences to display it
117 in color on the terminal.
120 Specify a file containing a list of defconfigs to move
123 Perform a trial run that does not make any changes. It is useful to
124 see what is going to happen before one actually runs it.
127 Exit immediately if Make exits with a non-zero status while processing
131 Do "make savedefconfig" forcibly for all the defconfig files.
132 If not specified, "make savedefconfig" only occurs for cases
133 where at least one CONFIG was moved.
136 Only cleanup the headers; skip the defconfig processing
139 Specify the number of threads to run simultaneously. If not specified,
140 the number of threads is the same as the number of CPU cores.
143 Specify the git ref to clone for building the autoconf.mk. If unspecified
144 use the CWD. This is useful for when changes to the Kconfig affect the
145 default values and you want to capture the state of the defconfig from
146 before that change was in effect. If in doubt, specify a ref pre-Kconfig
147 changes (use HEAD if Kconfig changes are not committed). Worst case it will
148 take a bit longer to run, but will always do the right thing.
151 Show any build errors as boards are built
153 To see the complete list of supported options, run
155 $ tools/moveconfig.py -h
163 import multiprocessing
173 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
176 # Here is the list of cross-tools I use.
177 # Most of them are available at kernel.org
178 # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
179 # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
180 # blackfin: http://sourceforge.net/projects/adi-toolchain/files/
181 # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
182 # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
183 # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
185 # openrisc kernel.org toolchain is out of date, download latest one from
186 # http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions
189 'aarch64': 'aarch64-linux-',
190 'arm': 'arm-unknown-linux-gnueabi-',
191 'avr32': 'avr32-linux-',
192 'blackfin': 'bfin-elf-',
193 'm68k': 'm68k-linux-',
194 'microblaze': 'microblaze-linux-',
195 'mips': 'mips-linux-',
196 'nds32': 'nds32le-linux-',
197 'nios2': 'nios2-linux-gnu-',
198 'openrisc': 'or1k-elf-',
199 'powerpc': 'powerpc-linux-',
200 'sh': 'sh-linux-gnu-',
201 'sparc': 'sparc-linux-',
202 'x86': 'i386-linux-',
203 'xtensa': 'xtensa-linux-'
209 STATE_SAVEDEFCONFIG = 3
220 COLOR_PURPLE = '0;35'
222 COLOR_LIGHT_GRAY = '0;37'
223 COLOR_DARK_GRAY = '1;30'
224 COLOR_LIGHT_RED = '1;31'
225 COLOR_LIGHT_GREEN = '1;32'
226 COLOR_YELLOW = '1;33'
227 COLOR_LIGHT_BLUE = '1;34'
228 COLOR_LIGHT_PURPLE = '1;35'
229 COLOR_LIGHT_CYAN = '1;36'
232 ### helper functions ###
234 """Get the file object of '/dev/null' device."""
236 devnull = subprocess.DEVNULL # py3k
237 except AttributeError:
238 devnull = open(os.devnull, 'wb')
241 def check_top_directory():
242 """Exit if we are not at the top of source directory."""
243 for f in ('README', 'Licenses'):
244 if not os.path.exists(f):
245 sys.exit('Please run at the top of source directory.')
247 def check_clean_directory():
248 """Exit if the source tree is not clean."""
249 for f in ('.config', 'include/config'):
250 if os.path.exists(f):
251 sys.exit("source tree is not clean, please run 'make mrproper'")
254 """Get the command name of GNU Make.
256 U-Boot needs GNU Make for building, but the command name is not
257 necessarily "make". (for example, "gmake" on FreeBSD).
258 Returns the most appropriate command name on your system.
260 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
261 ret = process.communicate()
262 if process.returncode:
263 sys.exit('GNU Make not found')
264 return ret[0].rstrip()
266 def get_all_defconfigs():
267 """Get all the defconfig files under the configs/ directory."""
269 for (dirpath, dirnames, filenames) in os.walk('configs'):
270 dirpath = dirpath[len('configs') + 1:]
271 for filename in fnmatch.filter(filenames, '*_defconfig'):
272 defconfigs.append(os.path.join(dirpath, filename))
276 def color_text(color_enabled, color, string):
277 """Return colored string."""
279 # LF should not be surrounded by the escape sequence.
280 # Otherwise, additional whitespace or line-feed might be printed.
281 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
282 for s in string.split('\n') ])
286 def show_diff(a, b, file_path, color_enabled):
287 """Show unidified diff.
290 a: A list of lines (before)
291 b: A list of lines (after)
292 file_path: Path to the file
293 color_enabled: Display the diff in color
296 diff = difflib.unified_diff(a, b,
297 fromfile=os.path.join('a', file_path),
298 tofile=os.path.join('b', file_path))
301 if line[0] == '-' and line[1] != '-':
302 print color_text(color_enabled, COLOR_RED, line),
303 elif line[0] == '+' and line[1] != '+':
304 print color_text(color_enabled, COLOR_GREEN, line),
308 def update_cross_compile(color_enabled):
309 """Update per-arch CROSS_COMPILE via environment variables
311 The default CROSS_COMPILE values are available
312 in the CROSS_COMPILE list above.
314 You can override them via environment variables
315 CROSS_COMPILE_{ARCH}.
317 For example, if you want to override toolchain prefixes
318 for ARM and PowerPC, you can do as follows in your shell:
320 export CROSS_COMPILE_ARM=...
321 export CROSS_COMPILE_POWERPC=...
323 Then, this function checks if specified compilers really exist in your
328 for arch in os.listdir('arch'):
329 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
332 # arm64 is a special case
333 archs.append('aarch64')
336 env = 'CROSS_COMPILE_' + arch.upper()
337 cross_compile = os.environ.get(env)
338 if not cross_compile:
339 cross_compile = CROSS_COMPILE.get(arch, '')
341 for path in os.environ["PATH"].split(os.pathsep):
342 gcc_path = os.path.join(path, cross_compile + 'gcc')
343 if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
346 print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
347 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
348 % (cross_compile, arch))
351 CROSS_COMPILE[arch] = cross_compile
353 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
355 """Extend matched lines if desired patterns are found before/after already
359 lines: A list of lines handled.
360 matched: A list of line numbers that have been already matched.
361 (will be updated by this function)
362 pre_patterns: A list of regular expression that should be matched as
364 post_patterns: A list of regular expression that should be matched as
366 extend_pre: Add the line number of matched preamble to the matched list.
367 extend_post: Add the line number of matched postamble to the matched list.
369 extended_matched = []
382 for p in pre_patterns:
383 if p.search(lines[i - 1]):
389 for p in post_patterns:
390 if p.search(lines[j]):
397 extended_matched.append(i - 1)
399 extended_matched.append(j)
401 matched += extended_matched
404 def cleanup_one_header(header_path, patterns, options):
405 """Clean regex-matched lines away from a file.
408 header_path: path to the cleaned file.
409 patterns: list of regex patterns. Any lines matching to these
410 patterns are deleted.
411 options: option flags.
413 with open(header_path) as f:
414 lines = f.readlines()
417 for i, line in enumerate(lines):
418 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
421 for pattern in patterns:
422 if pattern.search(line):
429 # remove empty #ifdef ... #endif, successive blank lines
430 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
431 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
432 pattern_endif = re.compile(r'#\s*endif\W') # #endif
433 pattern_blank = re.compile(r'^\s*$') # empty line
436 old_matched = copy.copy(matched)
437 extend_matched_lines(lines, matched, [pattern_if],
438 [pattern_endif], True, True)
439 extend_matched_lines(lines, matched, [pattern_elif],
440 [pattern_elif, pattern_endif], True, False)
441 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
442 [pattern_blank], False, True)
443 extend_matched_lines(lines, matched, [pattern_blank],
444 [pattern_elif, pattern_endif], True, False)
445 extend_matched_lines(lines, matched, [pattern_blank],
446 [pattern_blank], True, False)
447 if matched == old_matched:
450 tolines = copy.copy(lines)
452 for i in reversed(matched):
455 show_diff(lines, tolines, header_path, options.color)
460 with open(header_path, 'w') as f:
464 def cleanup_headers(configs, options):
465 """Delete config defines from board headers.
468 configs: A list of CONFIGs to remove.
469 options: option flags.
472 choice = raw_input('Clean up headers? [y/n]: ').lower()
474 if choice == 'y' or choice == 'n':
481 for config in configs:
482 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
483 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
485 for dir in 'include', 'arch', 'board':
486 for (dirpath, dirnames, filenames) in os.walk(dir):
487 if dirpath == os.path.join('include', 'generated'):
489 for filename in filenames:
490 if not fnmatch.fnmatch(filename, '*~'):
491 cleanup_one_header(os.path.join(dirpath, filename),
494 def cleanup_one_extra_option(defconfig_path, configs, options):
495 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
498 defconfig_path: path to the cleaned defconfig file.
499 configs: A list of CONFIGs to remove.
500 options: option flags.
503 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
506 with open(defconfig_path) as f:
507 lines = f.readlines()
509 for i, line in enumerate(lines):
510 if line.startswith(start) and line.endswith(end):
513 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
516 old_tokens = line[len(start):-len(end)].split(',')
519 for token in old_tokens:
520 pos = token.find('=')
521 if not (token[:pos] if pos >= 0 else token) in configs:
522 new_tokens.append(token)
524 if new_tokens == old_tokens:
527 tolines = copy.copy(lines)
530 tolines[i] = start + ','.join(new_tokens) + end
534 show_diff(lines, tolines, defconfig_path, options.color)
539 with open(defconfig_path, 'w') as f:
543 def cleanup_extra_options(configs, options):
544 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
547 configs: A list of CONFIGs to remove.
548 options: option flags.
551 choice = raw_input('Clean up CONFIG_SYS_EXTRA_OPTIONS? [y/n]: ').lower()
553 if choice == 'y' or choice == 'n':
559 configs = [ config[len('CONFIG_'):] for config in configs ]
561 defconfigs = get_all_defconfigs()
563 for defconfig in defconfigs:
564 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
570 """Progress Indicator"""
572 def __init__(self, total):
573 """Create a new progress indicator.
576 total: A number of defconfig files to process.
582 """Increment the number of processed defconfig files."""
587 """Display the progress."""
588 print ' %d defconfigs out of %d\r' % (self.current, self.total),
593 """A parser of .config and include/autoconf.mk."""
595 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
596 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
598 def __init__(self, configs, options, build_dir):
599 """Create a new parser.
602 configs: A list of CONFIGs to move.
603 options: option flags.
604 build_dir: Build directory.
606 self.configs = configs
607 self.options = options
608 self.dotconfig = os.path.join(build_dir, '.config')
609 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
610 self.config_autoconf = os.path.join(build_dir, 'include', 'config',
612 self.defconfig = os.path.join(build_dir, 'defconfig')
614 def get_cross_compile(self):
615 """Parse .config file and return CROSS_COMPILE.
618 A string storing the compiler prefix for the architecture.
619 Return a NULL string for architectures that do not require
620 compiler prefix (Sandbox and native build is the case).
621 Return None if the specified compiler is missing in your PATH.
622 Caller should distinguish '' and None.
626 for line in open(self.dotconfig):
627 m = self.re_arch.match(line)
631 m = self.re_cpu.match(line)
639 if arch == 'arm' and cpu == 'armv8':
642 return CROSS_COMPILE.get(arch, None)
644 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
645 """Parse .config, defconfig, include/autoconf.mk for one config.
647 This function looks for the config options in the lines from
648 defconfig, .config, and include/autoconf.mk in order to decide
649 which action should be taken for this defconfig.
652 config: CONFIG name to parse.
653 dotconfig_lines: lines from the .config file.
654 autoconf_lines: lines from the include/autoconf.mk file.
657 A tupple of the action for this defconfig and the line
658 matched for the config.
660 not_set = '# %s is not set' % config
662 for line in dotconfig_lines:
664 if line.startswith(config + '=') or line == not_set:
668 return (ACTION_NO_ENTRY, config)
670 for line in autoconf_lines:
672 if line.startswith(config + '='):
678 # If this CONFIG is neither bool nor trisate
679 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
680 # tools/scripts/define2mk.sed changes '1' to 'y'.
681 # This is a problem if the CONFIG is int type.
682 # Check the type in Kconfig and handle it correctly.
683 if new_val[-2:] == '=y':
684 new_val = new_val[:-1] + '1'
686 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
689 def update_dotconfig(self):
690 """Parse files for the config options and update the .config.
692 This function parses the generated .config and include/autoconf.mk
693 searching the target options.
694 Move the config option(s) to the .config as needed.
697 defconfig: defconfig name.
700 Return a tuple of (updated flag, log string).
701 The "updated flag" is True if the .config was updated, False
702 otherwise. The "log string" shows what happend to the .config.
708 with open(self.dotconfig) as f:
709 dotconfig_lines = f.readlines()
711 with open(self.autoconf) as f:
712 autoconf_lines = f.readlines()
714 for config in self.configs:
715 result = self.parse_one_config(config, dotconfig_lines,
717 results.append(result)
721 for (action, value) in results:
722 if action == ACTION_MOVE:
723 actlog = "Move '%s'" % value
724 log_color = COLOR_LIGHT_GREEN
725 elif action == ACTION_NO_ENTRY:
726 actlog = "%s is not defined in Kconfig. Do nothing." % value
727 log_color = COLOR_LIGHT_BLUE
728 elif action == ACTION_NO_CHANGE:
729 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
731 log_color = COLOR_LIGHT_PURPLE
733 sys.exit("Internal Error. This should not happen.")
735 log += color_text(self.options.color, log_color, actlog) + '\n'
737 with open(self.dotconfig, 'a') as f:
738 for (action, value) in results:
739 if action == ACTION_MOVE:
740 f.write(value + '\n')
743 self.results = results
744 os.remove(self.config_autoconf)
745 os.remove(self.autoconf)
747 return (updated, log)
749 def check_defconfig(self):
750 """Check the defconfig after savedefconfig
753 Return additional log if moved CONFIGs were removed again by
754 'make savedefconfig'.
759 with open(self.defconfig) as f:
760 defconfig_lines = f.readlines()
762 for (action, value) in self.results:
763 if action != ACTION_MOVE:
765 if not value + '\n' in defconfig_lines:
766 log += color_text(self.options.color, COLOR_YELLOW,
767 "'%s' was removed by savedefconfig.\n" %
774 """A slot to store a subprocess.
776 Each instance of this class handles one subprocess.
777 This class is useful to control multiple threads
778 for faster processing.
781 def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir):
782 """Create a new process slot.
785 configs: A list of CONFIGs to move.
786 options: option flags.
787 progress: A progress indicator.
788 devnull: A file object of '/dev/null'.
789 make_cmd: command name of GNU Make.
790 reference_src_dir: Determine the true starting config state from this
793 self.options = options
794 self.progress = progress
795 self.build_dir = tempfile.mkdtemp()
796 self.devnull = devnull
797 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
798 self.reference_src_dir = reference_src_dir
799 self.parser = KconfigParser(configs, options, self.build_dir)
800 self.state = STATE_IDLE
801 self.failed_boards = []
802 self.suspicious_boards = []
805 """Delete the working directory
807 This function makes sure the temporary directory is cleaned away
808 even if Python suddenly dies due to error. It should be done in here
809 because it is guaranteed the destructor is always invoked when the
810 instance of the class gets unreferenced.
812 If the subprocess is still running, wait until it finishes.
814 if self.state != STATE_IDLE:
815 while self.ps.poll() == None:
817 shutil.rmtree(self.build_dir)
819 def add(self, defconfig):
820 """Assign a new subprocess for defconfig and add it to the slot.
822 If the slot is vacant, create a new subprocess for processing the
823 given defconfig and add it to the slot. Just returns False if
824 the slot is occupied (i.e. the current subprocess is still running).
827 defconfig: defconfig name.
830 Return True on success or False on failure
832 if self.state != STATE_IDLE:
835 self.defconfig = defconfig
837 self.current_src_dir = self.reference_src_dir
842 """Check the status of the subprocess and handle it as needed.
844 Returns True if the slot is vacant (i.e. in idle state).
845 If the configuration is successfully finished, assign a new
846 subprocess to build include/autoconf.mk.
847 If include/autoconf.mk is generated, invoke the parser to
848 parse the .config and the include/autoconf.mk, moving
849 config options to the .config as needed.
850 If the .config was updated, run "make savedefconfig" to sync
851 it, update the original defconfig, and then set the slot back
855 Return True if the subprocess is terminated, False otherwise
857 if self.state == STATE_IDLE:
860 if self.ps.poll() == None:
863 if self.ps.poll() != 0:
865 elif self.state == STATE_DEFCONFIG:
866 if self.reference_src_dir and not self.current_src_dir:
867 self.do_savedefconfig()
870 elif self.state == STATE_AUTOCONF:
871 if self.current_src_dir:
872 self.current_src_dir = None
875 self.do_savedefconfig()
876 elif self.state == STATE_SAVEDEFCONFIG:
877 self.update_defconfig()
879 sys.exit("Internal Error. This should not happen.")
881 return True if self.state == STATE_IDLE else False
883 def handle_error(self):
884 """Handle error cases."""
886 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
887 "Failed to process.\n")
888 if self.options.verbose:
889 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
890 self.ps.stderr.read())
893 def do_defconfig(self):
894 """Run 'make <board>_defconfig' to create the .config file."""
896 cmd = list(self.make_cmd)
897 cmd.append(self.defconfig)
898 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
899 stderr=subprocess.PIPE,
900 cwd=self.current_src_dir)
901 self.state = STATE_DEFCONFIG
903 def do_autoconf(self):
904 """Run 'make include/config/auto.conf'."""
906 self.cross_compile = self.parser.get_cross_compile()
907 if self.cross_compile is None:
908 self.log += color_text(self.options.color, COLOR_YELLOW,
909 "Compiler is missing. Do nothing.\n")
913 cmd = list(self.make_cmd)
914 if self.cross_compile:
915 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
916 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
917 cmd.append('include/config/auto.conf')
918 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
919 stderr=subprocess.PIPE,
920 cwd=self.current_src_dir)
921 self.state = STATE_AUTOCONF
923 def do_savedefconfig(self):
924 """Update the .config and run 'make savedefconfig'."""
926 (updated, log) = self.parser.update_dotconfig()
929 if not self.options.force_sync and not updated:
933 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
934 "Syncing by savedefconfig...\n")
936 self.log += "Syncing by savedefconfig (forced by option)...\n"
938 cmd = list(self.make_cmd)
939 cmd.append('savedefconfig')
940 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
941 stderr=subprocess.PIPE)
942 self.state = STATE_SAVEDEFCONFIG
944 def update_defconfig(self):
945 """Update the input defconfig and go back to the idle state."""
947 log = self.parser.check_defconfig()
949 self.suspicious_boards.append(self.defconfig)
951 orig_defconfig = os.path.join('configs', self.defconfig)
952 new_defconfig = os.path.join(self.build_dir, 'defconfig')
953 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
956 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
957 "defconfig was updated.\n")
959 if not self.options.dry_run and updated:
960 shutil.move(new_defconfig, orig_defconfig)
963 def finish(self, success):
964 """Display log along with progress and go to the idle state.
967 success: Should be True when the defconfig was processed
968 successfully, or False when it fails.
970 # output at least 30 characters to hide the "* defconfigs out of *".
971 log = self.defconfig.ljust(30) + '\n'
973 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
974 # Some threads are running in parallel.
975 # Print log atomically to not mix up logs from different threads.
976 print >> (sys.stdout if success else sys.stderr), log
979 if self.options.exit_on_error:
980 sys.exit("Exit on error.")
981 # If --exit-on-error flag is not set, skip this board and continue.
982 # Record the failed board.
983 self.failed_boards.append(self.defconfig)
987 self.state = STATE_IDLE
989 def get_failed_boards(self):
990 """Returns a list of failed boards (defconfigs) in this slot.
992 return self.failed_boards
994 def get_suspicious_boards(self):
995 """Returns a list of boards (defconfigs) with possible misconversion.
997 return self.suspicious_boards
1001 """Controller of the array of subprocess slots."""
1003 def __init__(self, configs, options, progress, reference_src_dir):
1004 """Create a new slots controller.
1007 configs: A list of CONFIGs to move.
1008 options: option flags.
1009 progress: A progress indicator.
1010 reference_src_dir: Determine the true starting config state from this
1013 self.options = options
1015 devnull = get_devnull()
1016 make_cmd = get_make_cmd()
1017 for i in range(options.jobs):
1018 self.slots.append(Slot(configs, options, progress, devnull,
1019 make_cmd, reference_src_dir))
1021 def add(self, defconfig):
1022 """Add a new subprocess if a vacant slot is found.
1025 defconfig: defconfig name to be put into.
1028 Return True on success or False on failure
1030 for slot in self.slots:
1031 if slot.add(defconfig):
1035 def available(self):
1036 """Check if there is a vacant slot.
1039 Return True if at lease one vacant slot is found, False otherwise.
1041 for slot in self.slots:
1047 """Check if all slots are vacant.
1050 Return True if all the slots are vacant, False otherwise.
1053 for slot in self.slots:
1058 def show_failed_boards(self):
1059 """Display all of the failed boards (defconfigs)."""
1061 output_file = 'moveconfig.failed'
1063 for slot in self.slots:
1064 boards += slot.get_failed_boards()
1067 boards = '\n'.join(boards) + '\n'
1068 msg = "The following boards were not processed due to error:\n"
1070 msg += "(the list has been saved in %s)\n" % output_file
1071 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1074 with open(output_file, 'w') as f:
1077 def show_suspicious_boards(self):
1078 """Display all boards (defconfigs) with possible misconversion."""
1080 output_file = 'moveconfig.suspicious'
1082 for slot in self.slots:
1083 boards += slot.get_suspicious_boards()
1086 boards = '\n'.join(boards) + '\n'
1087 msg = "The following boards might have been converted incorrectly.\n"
1088 msg += "It is highly recommended to check them manually:\n"
1090 msg += "(the list has been saved in %s)\n" % output_file
1091 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1094 with open(output_file, 'w') as f:
1097 class ReferenceSource:
1099 """Reference source against which original configs should be parsed."""
1101 def __init__(self, commit):
1102 """Create a reference source directory based on a specified commit.
1105 commit: commit to git-clone
1107 self.src_dir = tempfile.mkdtemp()
1108 print "Cloning git repo to a separate work directory..."
1109 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1111 print "Checkout '%s' to build the original autoconf.mk." % \
1112 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1113 subprocess.check_output(['git', 'checkout', commit],
1114 stderr=subprocess.STDOUT, cwd=self.src_dir)
1117 """Delete the reference source directory
1119 This function makes sure the temporary directory is cleaned away
1120 even if Python suddenly dies due to error. It should be done in here
1121 because it is guaranteed the destructor is always invoked when the
1122 instance of the class gets unreferenced.
1124 shutil.rmtree(self.src_dir)
1127 """Return the absolute path to the reference source directory."""
1131 def move_config(configs, options):
1132 """Move config options to defconfig files.
1135 configs: A list of CONFIGs to move.
1136 options: option flags
1138 if len(configs) == 0:
1139 if options.force_sync:
1140 print 'No CONFIG is specified. You are probably syncing defconfigs.',
1142 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1144 print 'Move ' + ', '.join(configs),
1145 print '(jobs: %d)\n' % options.jobs
1148 reference_src = ReferenceSource(options.git_ref)
1149 reference_src_dir = reference_src.get_dir()
1151 reference_src_dir = None
1153 if options.defconfigs:
1154 defconfigs = [line.strip() for line in open(options.defconfigs)]
1155 for i, defconfig in enumerate(defconfigs):
1156 if not defconfig.endswith('_defconfig'):
1157 defconfigs[i] = defconfig + '_defconfig'
1158 if not os.path.exists(os.path.join('configs', defconfigs[i])):
1159 sys.exit('%s - defconfig does not exist. Stopping.' %
1162 defconfigs = get_all_defconfigs()
1164 progress = Progress(len(defconfigs))
1165 slots = Slots(configs, options, progress, reference_src_dir)
1167 # Main loop to process defconfig files:
1168 # Add a new subprocess into a vacant slot.
1169 # Sleep if there is no available slot.
1170 for defconfig in defconfigs:
1171 while not slots.add(defconfig):
1172 while not slots.available():
1173 # No available slot: sleep for a while
1174 time.sleep(SLEEP_TIME)
1176 # wait until all the subprocesses finish
1177 while not slots.empty():
1178 time.sleep(SLEEP_TIME)
1181 slots.show_failed_boards()
1182 slots.show_suspicious_boards()
1186 cpu_count = multiprocessing.cpu_count()
1187 except NotImplementedError:
1190 parser = optparse.OptionParser()
1192 parser.add_option('-c', '--color', action='store_true', default=False,
1193 help='display the log in color')
1194 parser.add_option('-d', '--defconfigs', type='string',
1195 help='a file containing a list of defconfigs to move')
1196 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1197 help='perform a trial run (show log with no changes)')
1198 parser.add_option('-e', '--exit-on-error', action='store_true',
1200 help='exit immediately on any error')
1201 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1202 help='force sync by savedefconfig')
1203 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1204 action='store_true', default=False,
1205 help='only cleanup the headers')
1206 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1207 help='the number of jobs to run simultaneously')
1208 parser.add_option('-r', '--git-ref', type='string',
1209 help='the git ref to clone for building the autoconf.mk')
1210 parser.add_option('-v', '--verbose', action='store_true', default=False,
1211 help='show any build errors as boards are built')
1212 parser.usage += ' CONFIG ...'
1214 (options, configs) = parser.parse_args()
1216 if len(configs) == 0 and not options.force_sync:
1217 parser.print_usage()
1220 # prefix the option name with CONFIG_ if missing
1221 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1222 for config in configs ]
1224 check_top_directory()
1226 if not options.cleanup_headers_only:
1227 check_clean_directory()
1228 update_cross_compile(options.color)
1229 move_config(configs, options)
1232 cleanup_headers(configs, options)
1233 cleanup_extra_options(configs, options)
1235 if __name__ == '__main__':