3 Michael Stapelberg <michael@i3wm.org>
6 This document explains the http://www.buildbot.net/[buildbot] setup we use to
7 provide up-to-date documentation and debian packages at http://build.i3wm.org/.
8 We publish these information so that our setup is well-documented (thus
9 decreasing future maintenance effort) and because it might be interesting for
14 What we are doing in i3 is called Continuous Integration (see
15 http://en.wikipedia.org/wiki/Continuous_integration): we publish the changes we
16 make on our local machines as often as possible. In order to maintain a
17 continuously high quality, each time any developer pushes changes to the
18 official git repository, a number of quality assurance tools start running
21 1. Latest documentation is generated and provided at
22 http://build.i3wm.org/docs/. This makes it easy to link to documentation for
23 features which are only in the current git version, not in the released
25 2. The source code is compiled and it is automatically posted to the IRC
26 channel whether there were any compiler warnings. While developers should
27 notice compiler warnings, this mechanism creates a bit of public pressure
28 ("Oh, Michael introduced warnings with this commit!"). More importantly,
29 since this mechanism builds a dist tarball and then compiles that tarball,
30 any changes to the source which would result in an uncompilable dist tarball
31 are immediately obvious. Therefore, we could cut a release from the current
32 git version at any point in time.
33 3. The clang static analyzer runs and the resulting report is provided at
34 http://build.i3wm.org/clang-analyze/. While every developer needs to compile
35 his code before committing, he doesn’t necessarily use clang (so we catch
36 build failures when using clang) and he also probably doesn’t run a static
37 analyzer as part of his normal workflow. By just being available without any
38 friction, this mechanism encourages developers to look at the report and fix
40 4. Debian (and Ubuntu) packages are built. This not only ensures that we don’t
41 change anything in the source code which would lead to an FTBFS (Fails To
42 Build From Source) when building a Debian package, it also goes a long way
43 to encourage end users to test i3. To remove the need and resource
44 requirements for them to compile their own version of i3 regularly, we
45 provide packages that integrate conveniently with a normal Debian system
46 (e.g. that are automatically upgraded).
50 Previously, I was unsatisfied with the current state of FOSS CI tools like
51 Jenkins, Tinderbox and others. They either seemed bloated, hard to use,
52 outdated or unappealing for some other reason.
54 Then I discovered buildbot and was impressed by its flexibility. It let me
55 implement everything I wanted from a CI tool and (in my opinion) it is
56 light-weight, easy to deploy and well maintained.
58 The only downside of buildbot is its configuration and documentation: You need
59 to spend quite a bit of time (I needed multiple days) until it works the way
60 you want it to and oftentimes, the documentation is far too sparse. This is one
61 of the reasons why I’m publishing the i3 setup.
65 See the next section for a complete, copy & pasteable configuration file. This
66 section covers the most important aspects without covering every line.
68 This document assumes you are running buildbot 0.8.6p1.
72 Since i3 uses a central git repository, we use the official buildbot
73 https://github.com/buildbot/buildbot/blob/master/master/contrib/git_buildbot.py[git
74 post-receive hook] that sends the change information to the buildbot master.
78 There are two things (called "builders" in buildbot-language) which happen
79 whenever a new change in the +next+ branch of i3 occurs:
81 1. The "docs" builder builds and uploads the latest documentation. This happens
82 directly from the git repository with a custom asciidoc configuration which
83 indicates that these docs refer to the git version. Therefore, this builder
84 does not benefit from having a dist tarball available (contrary to the other
87 2. The "dist" builder prepares a dist tarball and then triggers the remaining
88 builders. This ensures that building the dist tarball (an operation which
89 takes about one minute due to documentation generation) only happens once.
91 Here is the relevant configuration part:
94 ---------------------------------------------
97 c['schedulers'].append(SingleBranchScheduler(
100 treeStableTimer = 10,
101 builderNames = [ 'dist', 'docs' ],
104 c['schedulers'].append(Triggerable(
105 name = 'dist-tarball-done',
106 builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
108 ---------------------------------------------
110 === Building the dist tarball
112 This builder clones the i3 git repository and runs "make dist", which creates a
113 tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then
114 renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable
115 name in the next steps) and uploaded to the buildbot master (since we can have
116 multiple buildslaves, we cannot just let it rest on the buildslave that built
117 it). Afterwards, old dist tarballs are cleaned up and the remaining builders
120 *Building a dist tarball*:
121 ---------------------------------------------
124 f = factories['dist'] = BuildFactory()
126 # Check out the git repository.
129 # Fill the 'gitversion' property with the output of git describe --tags.
130 f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
132 # Build the dist tarball.
133 cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
135 # Rename the created tarball to a well-known name.
137 name = 'rename tarball',
138 command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'),
141 # Upload the dist tarball to the master (other factories download it later).
142 f.addStep(transfer.FileUpload(
143 slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
144 masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
147 # Cleanup old dist tarballs (everything older than tree days).
148 f.addStep(master.MasterShellCommand(
149 command = "find distballs -mtime +3 -exec rm '{}' \;",
150 name = 'cleanup old dist tarballs',
153 # Everything worked fine, now trigger compilation.
155 schedulerNames = [ 'dist-tarball-done' ],
156 copy_properties = [ 'gitversion' ],
158 ---------------------------------------------
160 Three things are noteworthy about this part of the configuration:
162 1. For convenience, we call each factory +f+ (just like the global buildbot
163 config uses +c+ for the top-level configuration) and add it to a dictionary.
164 Factories in that dictionary are later automatically configured for each
167 2. We have a shared step called +s_git+ so that we only have one location in
168 the configuration file where we specify the git repository URL and branch.
170 3. We have a custom function called +cmd+ which is a shortcut for defining a
171 +ShellCommand+ with +haltOnFailure=True+ (since each step is critical) and
172 +logEnviron=False+ (for brevity).
174 Here are their definitions:
177 ---------------------------------------------
178 def cmd(factory, **kwargs):
179 factory.addStep(ShellCommand(
180 haltOnFailure = True,
184 ---------------------------------------------
187 ---------------------------------------------
189 repourl = 'git://code.i3wm.org/i3',
192 # Check out the latest revision, not the one which caused this build.
193 alwaysUseLatest = True,
195 # We cannot use shallow because it breaks git describe --tags.
198 # Delete remnants of previous builds.
201 # Store checkouts in source/ and copy them over to build/ to save
205 ---------------------------------------------
207 === Compiling the dist tarball
209 For this builder to work, you obviously need to install all the
210 build-dependencies for your software on each buildslave. In the case of i3,
211 this can be done with +apt-get build-dep i3-wm+.
213 The compilation is pretty straight-forward since it uses the builtin +Compile+
214 step. We call +make+ with +-j4+ (we don’t have enough buildslaves to make
215 figuring out the amount of cores at build-time worthwhile) and +DEBUG=0+ to
216 simulate release build conditions. Also, we pass the preprocessor flag
217 +-D_FORTIFY_SOURCE=2+ and the compiler flags +-Wformat+ and +-Wformat-security+
218 to enable additional warnings.
220 *Compiling the dist tarball*:
221 ---------------------------------------------
222 f = factories['compile'] = BuildFactory()
223 unpack_dist_tarball(f)
225 command = [ 'make', 'DEBUG=0', '-j4' ],
226 warningPattern = '.*warning: ',
227 warnOnWarnings = True,
228 workdir = 'build/DIST',
230 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
231 'CFLAGS': '-Wformat -Wformat-security'
235 f.addStep(WarningsToIRC())
236 ---------------------------------------------
238 Again, we use custom functions (and a custom buildstep) to make our lives
239 easier. Here is the definition of unpack_dist_tarball which adds three steps to
240 the factory that download and unpack the dist tarball to the +DIST/+ directory:
242 *unpack_dist_tarball*:
243 ---------------------------------------------
244 def unpack_dist_tarball(factory):
245 factory.addStep(transfer.FileDownload(
246 mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
247 slavedest = 'dist.tar.bz2',
250 factory.addStep(slave.MakeDirectory(dir = 'build/DIST'))
253 name = 'unpack dist tarball',
254 command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
256 ---------------------------------------------
258 The +WarningsToIRC+ build step is a custom build step which sets a property
259 called "ircsuffix" that is used by our custom IRC bot. This is covered later in
260 more detail. This property gets set to a green or red message, depending on
261 whether there were any warnings:
264 ---------------------------------------------
265 class WarningsToIRC(buildstep.BuildStep):
267 warnings = self.getProperty("warnings-count")
268 if warnings is not None and int(warnings) > 0:
269 warnings = int(warnings) # just to be sure
270 self.setProperty("ircsuffix", ("\0037 with %d warning%s!" %
271 (warnings, "s" if warnings != 1 else "")))
273 self.setProperty("ircsuffix", "\0033 without warnings")
274 self.finished(SUCCESS)
275 ---------------------------------------------
277 === Static code analysis
279 For this builder to work, you additionally need the +clang+ compiler on each
280 buildslave: +apt-get install clang+.
282 This builder uses only custom functions which you already know by now. It runs
283 scan-build, then moves scan-build’s output from a date-based directory directly
284 into the +CLANG/+ directory and uploads that to the buildmaster.
286 On the buildmaster, a webserver is configured which has a symlink to
287 +/home/build/i3-master/htdocs/clang-analyze+ in its document root.
289 *static code analysis*:
290 ---------------------------------------------
291 f = factories['clang-analyze'] = BuildFactory()
292 unpack_dist_tarball(f)
298 '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
301 workdir = 'build/DIST',
304 # remove the subdirectory -- we always want to overwrite
305 cmd(f, command = 'mv CLANG/*/* CLANG/')
307 f.addStep(transfer.DirectoryUpload(
309 masterdest = 'htdocs/clang-analyze',
311 name = 'upload output',
314 f.addStep(ClangToIRC())
315 ---------------------------------------------
317 The +ClangToIRC+ custom step is even simpler than +WarningsToIRC+. It simply
318 sets the ircsuffix property to a static message:
321 ---------------------------------------------
322 class ClangToIRC(buildstep.BuildStep):
324 self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
325 self.finished(SUCCESS)
326 ---------------------------------------------
328 === Generating documentation
330 This builder is the one which is the least clean of all. It uses the Debian
331 packaging information to decide which docs to publish and which manpages to
332 generate. Additionally, it uses a for loop instead of calling a script. I
333 recommend including a script to do this in your repository instead.
335 Apart from these concerns, the builder is straight-forward: It clones the git
336 repository, generates the documentation and then uploads the documentation to
339 *Generating documentation*:
340 ---------------------------------------------
341 f = factories['docs'] = BuildFactory()
343 # Fill the 'gitversion' property with the output of git describe --tags.
344 f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
345 cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
346 cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
347 f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
348 cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
349 cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
351 f.addStep(transfer.DirectoryUpload(
352 slavesrc = 'COPY-DOCS',
353 masterdest = 'htdocs/docs-git',
355 name = 'upload docs'))
357 f.addStep(DocsToIRC())
358 ---------------------------------------------
360 Just as +ClangToIRC+, +DocsToIRC+ appends a static message:
363 ---------------------------------------------
364 class DocsToIRC(buildstep.BuildStep):
366 self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
367 self.finished(SUCCESS)
368 ---------------------------------------------
370 === Building Debian/Ubuntu packages
372 This is the most complex builder of all. It uses +pbuilder-dist+, +debchange+,
373 +dpkg-buildpackage+ and +reprepro+ to generate a Debian repository with a
374 cleanly compiled package for amd64 and i386. In order for it to work, you need
375 to install the following packages: +apt-get install devscripts dpkg-dev
376 reprepro ubuntu-dev-tools pbuilder+. Afterwards, you need to allow the user as
377 which the buildslave runs to execute pbuilder via sudo without needing a
378 password, so add a config file like this one:
381 ---------------------------------------------
382 echo 'build ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder' > /etc/sudoers.d/build
383 ---------------------------------------------
385 Then, as the user as which your buildslave runs, setup the pbuilder
386 environments (you only need to do this once):
388 *pbuilder preparation*:
389 ---------------------------------------------
390 sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64
391 sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386
392 pbuilder-sid-amd64 create
393 pbuilder-sid-i386 create
394 ---------------------------------------------
396 Also, you will need a GPG key to sign these packages.
398 The debian builder starts by unpacking the dist tarball, copying the Debian
399 packaging from git, creating an empty Debian repository with the
400 +i3-autobuild-keyring+ contents in it. It then adds a new changelog entry to
401 reflect the git version and the fact that this package was built automatically,
402 builds a source package with +dpkg-buildpackage+ and adds it to the repository.
403 Afterwards, it updates each pbuilder and builds binary packages for each
404 architecture (amd64 and i386). After adding the resulting packages to the
405 repository, it uploads the repository to the buildmaster:
408 ---------------------------------------------
409 distributions = [ 'sid-amd64', 'sid-i386' ]
412 f = factories['debian-packages'] = BuildFactory()
413 # We need the git repository for the Debian packaging.
415 unpack_dist_tarball(f)
416 cmd(f, name = 'copy packaging', command = "cp -r debian DIST/")
418 # Add a new changelog entry to have the git version in the package version.
420 name = 'update changelog',
421 workdir = 'build/DIST',
422 command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
427 command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
428 workdir = 'build/DIST',
431 for dist in distributions:
432 f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist))
434 # Create debian sid repository
435 f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf'))
436 f.addStep(transfer.StringDownload(
439 Architectures: i386 amd64 source
441 DebIndices: Packages Release . .gz .bz2
442 DscIndices: Sources Release . .gz .bz2
443 SignWith: %(gpg_key)s
444 """ % { "gpg_key": gpg_key },
445 slavedest = 'REPO-sid/conf/distributions',
448 # add source package to repository
449 reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
451 # Add keyring to the repository. We need to run git clone on our own because
452 # the Git() step assumes there’s precisely one repository we want to deal with.
453 # No big deal since the i3-autobuild-keyring repository is not big.
455 name = 'clone keyring repo',
456 command = 'git clone git://code.i3wm.org/i3-autobuild-keyring',
458 reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
460 for dist in distributions:
461 # update the pbuilder
462 cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
464 # build the package for each dist
465 f.addStep(ShellCommand(
467 name = 'pkg ' + dist,
468 command = 'pbuilder-' + dist + ' build --binary-arch \
469 --buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
473 reprepro_include(f, 'RESULT-' + dist + '/*.changes')
475 # upload the sid repo
476 # Since the next step is cleaning up old files, we set haltOnFailure=True -- we
477 # prefer providing old packages over providing no packages at all :).
478 for directory in [ 'pool', 'dists' ]:
479 f.addStep(transfer.DirectoryUpload(
480 slavesrc = 'REPO-sid/' + directory,
481 masterdest = 'htdocs/debian/sid/' + directory,
483 name = 'upload sid ' + directory,
484 haltOnFailure = True,
487 f.addStep(master.MasterShellCommand(
488 command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
489 name = 'cleanup old packages',
492 # We ensure there is an empty i18n/Index to speed up apt (so that it does not
493 # try to download Translation-*)
494 f.addStep(master.MasterShellCommand(
495 command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
496 name = 'create i18n folder',
498 f.addStep(master.MasterShellCommand(
499 command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
500 name = 'touch i18n/Index',
502 ---------------------------------------------
504 The +reprepro_include+ command is defined as follows:
507 ---------------------------------------------
508 def reprepro_include(factory, path, debtype='deb', **kwargs):
510 name = 'reprepro include',
511 command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
514 ---------------------------------------------
516 Running such a builder for Ubuntu works exactly the same way, but you need to
517 replace "sid" with "precise" in all places (see the full configuration file for
522 We don’t advertise the HTTP status target. Instead, status is posted to IRC via
523 a custom bot. This bot provides an HTTP end point and buildbot is configured to
524 push status changes to that endpoint:
526 *http status target*:
527 ---------------------------------------------
528 c['status'].append(buildbot.status.status_push.HttpStatusPush(
529 serverUrl = 'http://localhost:8080/push_buildbot',
531 ---------------------------------------------
533 You can find the source code of that bot at
534 http://code.stapelberg.de/git/go-buildbot-announce/. As the name suggests, it
535 is written in Go. Also, it is quite specific to i3, so you might be better off
536 implementing such a bot (or plugin) on your own. It might make for a nice
537 example, though, especially back when its only feature was announcing the build
540 http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0
542 === Creating the buildslave
544 One more thing to note is that when creating the buildslave, you should use the
545 +--umask+ argument to configure the umask for all generated files:
547 *Creating the buildslave*:
548 --------------------------------------------------------------------------------------
549 buildslave create-slave --umask=022 i3-buildslave buildbot.i3wm.org build-1 <password>
550 --------------------------------------------------------------------------------------
552 == Full configuration file
554 This is the full configuration file, as tested and currently in use (except for
555 the passwords, though):
558 ---------------------------------------------
561 # vim:ts=4:sw=4:expandtab:syntax=python
563 # i3 buildbot configuration
564 # © 2012 Michael Stapelberg, Public Domain
565 # see http://i3wm.org/docs/buildbot.html for more information.
567 from buildbot.buildslave import BuildSlave
568 from buildbot.changes import pb
569 from buildbot.schedulers.basic import SingleBranchScheduler
570 from buildbot.schedulers.triggerable import Triggerable
571 from buildbot.process.properties import WithProperties
572 from buildbot.process.factory import BuildFactory
573 from buildbot.steps.source.git import Git
574 from buildbot.steps.shell import ShellCommand
575 from buildbot.steps.shell import Compile
576 from buildbot.steps.trigger import Trigger
577 from buildbot.steps import shell, transfer, master, slave
578 from buildbot.config import BuilderConfig
579 from buildbot.process import buildstep
580 from buildbot.status import html
581 from buildbot.status import words
582 import buildbot.status.status_push
583 from buildbot.status.web import auth, authz
584 from buildbot.status.builder import SUCCESS, FAILURE
586 c = BuildmasterConfig = {}
588 c['slaves'] = [BuildSlave('docsteel-vm', 'secret')]
589 c['slavePortnum'] = 9989
590 # Changes are pushed to buildbot using a git hook.
591 c['change_source'] = [pb.PBChangeSource(
596 ################################################################################
598 ################################################################################
602 # The first scheduler kicks off multiple builders:
603 # • 'dist' builds a dist tarball and starts the triggerable schedulers
605 # • 'docs' builds the documentation with a special asciidoc configuration
606 # (therefore, it does not profit from a dist tarball and can be run in
608 c['schedulers'].append(SingleBranchScheduler(
611 treeStableTimer = 10,
612 builderNames = [ 'dist', 'docs' ],
615 c['schedulers'].append(Triggerable(
616 name = 'dist-tarball-done',
617 builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
620 ################################################################################
621 # Shortcuts for builders
622 ################################################################################
624 # shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False
625 def cmd(factory, **kwargs):
626 factory.addStep(ShellCommand(
632 # Shortcut to add steps necessary to download and unpack the dist tarball.
633 def unpack_dist_tarball(factory):
634 factory.addStep(transfer.FileDownload(
635 mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
636 slavedest='dist.tar.bz2',
638 factory.addStep(slave.MakeDirectory(dir='build/DIST'))
640 name = 'unpack dist tarball',
641 command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
644 # Includes the given path in REPO-sid using reprepro.
645 def reprepro_include(factory, path, debtype='deb', **kwargs):
647 name = 'reprepro include',
648 command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
652 def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs):
654 name = 'reprepro include',
655 command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path,
659 ################################################################################
661 ################################################################################
663 # Adds the ircsuffix property to reflect whether there were warnings.
664 class WarningsToIRC(buildstep.BuildStep):
666 warnings = self.getProperty("warnings-count")
667 if warnings is not None and int(warnings) > 0:
668 warnings = int(warnings) # just to be sure
669 self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else ""))
671 self.setProperty("ircsuffix", "\0033 without warnings")
672 self.finished(SUCCESS)
674 # Adds a link to the automatically generated documentation.
675 class DocsToIRC(buildstep.BuildStep):
677 self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
678 self.finished(SUCCESS)
680 # Adds a link to the clang report.
681 class ClangToIRC(buildstep.BuildStep):
683 self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
684 self.finished(SUCCESS)
686 ################################################################################
687 # Shared steps, used in different factories.
688 ################################################################################
691 repourl='git://code.i3wm.org/i3',
694 # Check out the latest revision, not the one which caused this build.
695 alwaysUseLatest=True,
697 # We cannot use shallow because it breaks git describe --tags.
700 # Delete remnants of previous builds.
703 # Store checkouts in source/ and copy them over to build/ to save
707 # XXX: In newer versions of buildbot (> 0.8.6), we want to use
708 # getDescription={ 'tags': True } here and get rid of the extra git
709 # describe --tags step.
712 ################################################################################
713 # factory: "dist" — builds the dist tarball once (used by all other factories)
714 ################################################################################
718 f = factories['dist'] = BuildFactory()
719 # Check out the git repository.
721 # Fill the 'gitversion' property with the output of git describe --tags.
722 f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
723 # Build the dist tarball.
724 cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
725 # Rename the created tarball to a well-known name.
726 cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'))
727 # Upload the dist tarball to the master (other factories download it later).
728 f.addStep(transfer.FileUpload(
729 slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
730 masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
732 # Cleanup old dist tarballs (everything older than tree days).
733 f.addStep(master.MasterShellCommand(
734 command = "find distballs -mtime +3 -exec rm '{}' \;",
735 name = 'cleanup old dist tarballs',
737 # Everything worked fine, now trigger compilation.
739 schedulerNames = [ 'dist-tarball-done' ],
740 copy_properties = [ 'gitversion' ],
743 ################################################################################
744 # factory: "compile" — compiles the dist tarball and reports warnings
745 ################################################################################
747 f = factories['compile'] = BuildFactory()
748 unpack_dist_tarball(f)
750 command = [ 'make', 'DEBUG=0', '-j4' ],
751 warningPattern = '.*warning: ',
752 warnOnWarnings = True,
753 workdir = 'build/DIST',
755 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
756 'CFLAGS': '-Wformat -Wformat-security'
760 f.addStep(WarningsToIRC())
762 ################################################################################
763 # factory: "clang-analyze" — runs a static code analysis
764 ################################################################################
765 # $ sudo apt-get install clang
767 f = factories['clang-analyze'] = BuildFactory()
768 unpack_dist_tarball(f)
774 '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
777 workdir = 'build/DIST',
780 # remove the subdirectory -- we always want to overwrite
781 cmd(f, command = 'mv CLANG/*/* CLANG/')
783 f.addStep(transfer.DirectoryUpload(
785 masterdest = 'htdocs/clang-analyze',
787 name = 'upload output',
790 f.addStep(ClangToIRC())
792 ################################################################################
793 # factory: "docs" — builds documentation with a special asciidoc conf
794 ################################################################################
796 f = factories['docs'] = BuildFactory()
798 # Fill the 'gitversion' property with the output of git describe --tags.
799 f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
800 cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
801 cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
802 f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
803 cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
804 cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
806 f.addStep(transfer.DirectoryUpload(
807 slavesrc = 'COPY-DOCS',
808 masterdest = 'htdocs/docs-git',
810 name = 'upload docs'))
812 f.addStep(DocsToIRC())
814 ################################################################################
815 # factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386
816 ################################################################################
818 distributions = [ 'sid-amd64', 'sid-i386' ]
821 f = factories['debian-packages'] = BuildFactory()
822 # We need the git repository for the Debian packaging.
824 unpack_dist_tarball(f)
825 cmd(f, name='copy packaging', command = "cp -r debian DIST/")
827 # Add a new changelog entry to have the git version in the package version.
829 name = 'update changelog',
830 workdir = 'build/DIST',
831 command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
836 command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
837 workdir = 'build/DIST',
840 for dist in distributions:
841 f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
843 # Create debian sid repository
844 f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
845 f.addStep(transfer.StringDownload(
848 Architectures: i386 amd64 source
850 DebIndices: Packages Release . .gz .bz2
851 DscIndices: Sources Release . .gz .bz2
852 SignWith: %(gpg_key)s
853 """ % { "gpg_key": gpg_key },
854 slavedest = 'REPO-sid/conf/distributions',
857 # add source package to repository
858 reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
860 # Add keyring to the repository. We need to run git clone on our own because
861 # the Git() step assumes there’s precisely one repository we want to deal with.
862 # No big deal since the i3-autobuild-keyring repository is not big.
863 cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
864 reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
866 for dist in distributions:
867 # update the pbuilder
868 cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
870 # build the package for each dist
871 f.addStep(ShellCommand(
873 name = 'pkg ' + dist,
874 command = 'pbuilder-' + dist + ' build --binary-arch \
875 --buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
879 reprepro_include(f, 'RESULT-' + dist + '/*.changes')
881 # upload the sid repo
882 # Since the next step is cleaning up old files, we set haltOnFailure=True -- we
883 # prefer providing old packages over providing no packages at all :).
884 for directory in [ 'pool', 'dists' ]:
885 f.addStep(transfer.DirectoryUpload(
886 slavesrc = 'REPO-sid/' + directory,
887 masterdest = 'htdocs/debian/sid/' + directory,
889 name = 'upload sid ' + directory,
890 haltOnFailure = True,
893 f.addStep(master.MasterShellCommand(
894 command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
895 name = 'cleanup old packages',
898 # We ensure there is an empty i18n/Index to speed up apt (so that it does not
899 # try to download Translation-*)
900 f.addStep(master.MasterShellCommand(
901 command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
902 name = 'create i18n folder',
904 f.addStep(master.MasterShellCommand(
905 command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
906 name = 'touch i18n/Index',
909 ################################################################################
910 # factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386
911 ################################################################################
913 distributions = [ 'precise-amd64', 'precise-i386' ]
916 f = factories['ubuntu-packages'] = BuildFactory()
917 # We need the git repository for the Debian packaging.
919 unpack_dist_tarball(f)
920 cmd(f, name='copy packaging', command = "cp -r debian DIST/")
922 # Add a new changelog entry to have the git version in the package version.
924 name = 'update changelog',
925 workdir = 'build/DIST',
926 command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
931 command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
932 workdir = 'build/DIST',
935 for dist in distributions:
936 f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
938 # Create debian sid repository
939 f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
940 f.addStep(transfer.StringDownload(
943 Architectures: i386 amd64 source
945 DebIndices: Packages Release . .gz .bz2
946 DscIndices: Sources Release . .gz .bz2
947 SignWith: %(gpg_key)s
948 """ % { "gpg_key": gpg_key },
949 slavedest = 'REPO-sid/conf/distributions',
952 # add source package to repository
953 reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc')
955 # Add keyring to the repository. We need to run git clone on our own because
956 # the Git() step assumes there’s precisely one repository we want to deal with.
957 # No big deal since the i3-autobuild-keyring repository is not big.
958 cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
959 reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes')
961 for dist in distributions:
962 # update the pbuilder
963 cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
965 # build the package for each dist
966 f.addStep(ShellCommand(
968 name = 'pkg ' + dist,
969 command = 'pbuilder-' + dist + ' build --binary-arch \
970 --buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
974 reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes')
976 # upload the sid repo
977 # Since the next step is cleaning up old files, we set haltOnFailure=True -- we
978 # prefer providing old packages over providing no packages at all :).
979 for directory in [ 'pool', 'dists' ]:
980 f.addStep(transfer.DirectoryUpload(
981 slavesrc = 'REPO-sid/' + directory,
982 masterdest = 'htdocs/ubuntu/precise/' + directory,
984 name = 'upload precise ' + directory,
985 haltOnFailure = True,
988 f.addStep(master.MasterShellCommand(
989 command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;",
990 name = 'cleanup old packages',
993 # We ensure there is an empty i18n/Index to speed up apt (so that it does not
994 # try to download Translation-*)
995 f.addStep(master.MasterShellCommand(
996 command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ],
997 name = 'create i18n folder',
999 f.addStep(master.MasterShellCommand(
1000 command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ],
1001 name = 'touch i18n/Index',
1007 # Add all builders to all buildslaves.
1008 for factoryname in factories.keys():
1009 c['builders'].append(BuilderConfig(
1011 slavenames=['docsteel-vm'],
1012 factory=factories[factoryname],
1016 ####### STATUS TARGETS
1020 authz_cfg=authz.Authz(
1021 gracefulShutdown = False,
1023 forceAllBuilds = False,
1024 pingBuilder = False,
1026 stopAllBuilds = False,
1027 cancelPendingBuild = False,
1030 c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
1032 c['status'].append(buildbot.status.status_push.HttpStatusPush(
1033 serverUrl = 'http://localhost:8080/push_buildbot',
1036 ####### PROJECT IDENTITY
1039 c['titleURL'] = 'http://i3wm.org/'
1040 # Removed so that search engines don’t crawl it
1041 c['buildbotURL'] = 'http://localhost/'
1046 # This specifies what database buildbot uses to store its state. You can leave
1047 # this at its default for all but the largest installations.
1048 'db_url' : "sqlite:///state.sqlite",
1050 ---------------------------------------------