--- /dev/null
+The i3 buildbot setup
+=====================
+Michael Stapelberg <michael@i3wm.org>
+August 2012
+
+This document explains the http://www.buildbot.net/[buildbot] setup we use to
+provide up-to-date documentation and debian packages at http://build.i3wm.org/.
+We publish these information so that our setup is well-documented (thus
+decreasing future maintenance effort) and because it might be interesting for
+other projects.
+
+== Introduction
+
+What we are doing in i3 is called Continuous Integration (see
+http://en.wikipedia.org/wiki/Continuous_integration): we publish the changes we
+make on our local machines as often as possible. In order to maintain a
+continuously high quality, each time any developer pushes changes to the
+official git repository, a number of quality assurance tools start running
+automatically:
+
+1. Latest documentation is generated and provided at
+ http://build.i3wm.org/docs/. This makes it easy to link to documentation for
+ features which are only in the current git version, not in the released
+ version.
+2. The source code is compiled and it is automatically posted to the IRC
+ channel whether there were any compiler warnings. While developers should
+ notice compiler warnings, this mechanism creates a bit of public pressure
+ ("Oh, Michael introduced warnings with this commit!"). More importantly,
+ since this mechanism builds a dist tarball and then compiles that tarball,
+ any changes to the source which would result in an uncompilable dist tarball
+ are immediately obvious. Therefore, we could cut a release from the current
+ git version at any point in time.
+3. The clang static analyzer runs and the resulting report is provided at
+ http://build.i3wm.org/clang-analyze/. While every developer needs to compile
+ his code before committing, he doesn’t necessarily use clang (so we catch
+ build failures when using clang) and he also probably doesn’t run a static
+ analyzer as part of his normal workflow. By just being available without any
+ friction, this mechanism encourages developers to look at the report and fix
+ problems.
+4. Debian (and Ubuntu) packages are built. This not only ensures that we don’t
+ change anything in the source code which would lead to an FTBFS (Fails To
+ Build From Source) when building a Debian package, it also goes a long way
+ to encourage end users to test i3. To remove the need and resource
+ requirements for them to compile their own version of i3 regularly, we
+ provide packages that integrate conveniently with a normal Debian system
+ (e.g. that are automatically upgraded).
+
+== Why buildbot?
+
+Previously, I was unsatisfied with the current state of FOSS CI tools like
+Jenkins, Tinderbox and others. They either seemed bloated, hard to use,
+outdated or unappealing for some other reason.
+
+Then I discovered buildbot and was impressed by its flexibility. It let me
+implement everything I wanted from a CI tool and (in my opinion) it is
+light-weight, easy to deploy and well maintained.
+
+The only downside of buildbot is its configuration and documentation: You need
+to spend quite a bit of time (I needed multiple days) until it works the way
+you want it to and oftentimes, the documentation is far too sparse. This is one
+of the reasons why I’m publishing the i3 setup.
+
+== Configuration
+
+See the next section for a complete, copy & pasteable configuration file. This
+section covers the most important aspects without covering every line.
+
+This document assumes you are running buildbot 0.8.6p1.
+
+=== Change sources
+
+Since i3 uses a central git repository, we use the official buildbot
+https://github.com/buildbot/buildbot/blob/master/master/contrib/git_buildbot.py[git
+post-receive hook] that sends the change information to the buildbot master.
+
+=== Schedulers
+
+There are two things (called "builders" in buildbot-language) which happen
+whenever a new change in the +next+ branch of i3 occurs:
+
+1. The "docs" builder builds and uploads the latest documentation. This happens
+ directly from the git repository with a custom asciidoc configuration which
+ indicates that these docs refer to the git version. Therefore, this builder
+ does not benefit from having a dist tarball available (contrary to the other
+ builders).
+
+2. The "dist" builder prepares a dist tarball and then triggers the remaining
+ builders. This ensures that building the dist tarball (an operation which
+ takes about one minute due to documentation generation) only happens once.
+
+Here is the relevant configuration part:
+
+*Schedulers*:
+---------------------------------------------
+c['schedulers'] = []
+
+c['schedulers'].append(SingleBranchScheduler(
+ name = 'dist',
+ branch = 'next',
+ treeStableTimer = 10,
+ builderNames = [ 'dist', 'docs' ],
+))
+
+c['schedulers'].append(Triggerable(
+ name = 'dist-tarball-done',
+ builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
+))
+---------------------------------------------
+
+=== Building the dist tarball
+
+This builder clones the i3 git repository and runs "make dist", which creates a
+tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then
+renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable
+name in the next steps) and uploaded to the buildbot master (since we can have
+multiple buildslaves, we cannot just let it rest on the buildslave that built
+it). Afterwards, old dist tarballs are cleaned up and the remaining builders
+are triggered:
+
+*Building a dist tarball*:
+---------------------------------------------
+factories = {}
+
+f = factories['dist'] = BuildFactory()
+
+# Check out the git repository.
+f.addStep(s_git)
+
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+
+# Build the dist tarball.
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
+
+# Rename the created tarball to a well-known name.
+cmd(f,
+ name = 'rename tarball',
+ command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'),
+)
+
+# Upload the dist tarball to the master (other factories download it later).
+f.addStep(transfer.FileUpload(
+ slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
+ masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+))
+
+# Cleanup old dist tarballs (everything older than tree days).
+f.addStep(master.MasterShellCommand(
+ command = "find distballs -mtime +3 -exec rm '{}' \;",
+ name = 'cleanup old dist tarballs',
+))
+
+# Everything worked fine, now trigger compilation.
+f.addStep(Trigger(
+ schedulerNames = [ 'dist-tarball-done' ],
+ copy_properties = [ 'gitversion' ],
+))
+---------------------------------------------
+
+Three things are noteworthy about this part of the configuration:
+
+1. For convenience, we call each factory +f+ (just like the global buildbot
+ config uses +c+ for the top-level configuration) and add it to a dictionary.
+ Factories in that dictionary are later automatically configured for each
+ buildslave.
+
+2. We have a shared step called +s_git+ so that we only have one location in
+ the configuration file where we specify the git repository URL and branch.
+
+3. We have a custom function called +cmd+ which is a shortcut for defining a
+ +ShellCommand+ with +haltOnFailure=True+ (since each step is critical) and
+ +logEnviron=False+ (for brevity).
+
+Here are their definitions:
+
+*cmd*:
+---------------------------------------------
+def cmd(factory, **kwargs):
+ factory.addStep(ShellCommand(
+ haltOnFailure = True,
+ logEnviron = False,
+ **kwargs
+ ))
+---------------------------------------------
+
+*s_git*:
+---------------------------------------------
+s_git = Git(
+ repourl = 'git://code.i3wm.org/i3',
+ branch = 'next',
+
+ # Check out the latest revision, not the one which caused this build.
+ alwaysUseLatest = True,
+
+ # We cannot use shallow because it breaks git describe --tags.
+ shallow = False,
+
+ # Delete remnants of previous builds.
+ mode = 'full',
+
+ # Store checkouts in source/ and copy them over to build/ to save
+ # bandwidth.
+ method = 'copy',
+)
+---------------------------------------------
+
+=== Compiling the dist tarball
+
+For this builder to work, you obviously need to install all the
+build-dependencies for your software on each buildslave. In the case of i3,
+this can be done with +apt-get build-dep i3-wm+.
+
+The compilation is pretty straight-forward since it uses the builtin +Compile+
+step. We call +make+ with +-j4+ (we don’t have enough buildslaves to make
+figuring out the amount of cores at build-time worthwhile) and +DEBUG=0+ to
+simulate release build conditions. Also, we pass the preprocessor flag
++-D_FORTIFY_SOURCE=2+ and the compiler flags +-Wformat+ and +-Wformat-security+
+to enable additional warnings.
+
+*Compiling the dist tarball*:
+---------------------------------------------
+f = factories['compile'] = BuildFactory()
+unpack_dist_tarball(f)
+f.addStep(Compile(
+ command = [ 'make', 'DEBUG=0', '-j4' ],
+ warningPattern = '.*warning: ',
+ warnOnWarnings = True,
+ workdir = 'build/DIST',
+ env = {
+ 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
+ 'CFLAGS': '-Wformat -Wformat-security'
+ },
+))
+
+f.addStep(WarningsToIRC())
+---------------------------------------------
+
+Again, we use custom functions (and a custom buildstep) to make our lives
+easier. Here is the definition of unpack_dist_tarball which adds three steps to
+the factory that download and unpack the dist tarball to the +DIST/+ directory:
+
+*unpack_dist_tarball*:
+---------------------------------------------
+def unpack_dist_tarball(factory):
+ factory.addStep(transfer.FileDownload(
+ mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+ slavedest = 'dist.tar.bz2',
+ ))
+
+ factory.addStep(slave.MakeDirectory(dir = 'build/DIST'))
+
+ cmd(factory,
+ name = 'unpack dist tarball',
+ command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
+ )
+---------------------------------------------
+
+The +WarningsToIRC+ build step is a custom build step which sets a property
+called "ircsuffix" that is used by our custom IRC bot. This is covered later in
+more detail. This property gets set to a green or red message, depending on
+whether there were any warnings:
+
+*WarningsToIRC*:
+---------------------------------------------
+class WarningsToIRC(buildstep.BuildStep):
+ def start(self):
+ warnings = self.getProperty("warnings-count")
+ if warnings is not None and int(warnings) > 0:
+ warnings = int(warnings) # just to be sure
+ self.setProperty("ircsuffix", ("\0037 with %d warning%s!" %
+ (warnings, "s" if warnings != 1 else "")))
+ else:
+ self.setProperty("ircsuffix", "\0033 without warnings")
+ self.finished(SUCCESS)
+---------------------------------------------
+
+=== Static code analysis
+
+For this builder to work, you additionally need the +clang+ compiler on each
+buildslave: +apt-get install clang+.
+
+This builder uses only custom functions which you already know by now. It runs
+scan-build, then moves scan-build’s output from a date-based directory directly
+into the +CLANG/+ directory and uploads that to the buildmaster.
+
+On the buildmaster, a webserver is configured which has a symlink to
++/home/build/i3-master/htdocs/clang-analyze+ in its document root.
+
+*static code analysis*:
+---------------------------------------------
+f = factories['clang-analyze'] = BuildFactory()
+unpack_dist_tarball(f)
+cmd(f,
+ name='analyze',
+ command = [
+ 'scan-build',
+ '-o', '../CLANG',
+ '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
+ 'make', '-j8',
+ ],
+ workdir = 'build/DIST',
+)
+
+# remove the subdirectory -- we always want to overwrite
+cmd(f, command = 'mv CLANG/*/* CLANG/')
+
+f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'CLANG',
+ masterdest = 'htdocs/clang-analyze',
+ compress = 'bz2',
+ name = 'upload output',
+))
+
+f.addStep(ClangToIRC())
+---------------------------------------------
+
+The +ClangToIRC+ custom step is even simpler than +WarningsToIRC+. It simply
+sets the ircsuffix property to a static message:
+
+*ClangToIRC*:
+---------------------------------------------
+class ClangToIRC(buildstep.BuildStep):
+ def start(self):
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
+ self.finished(SUCCESS)
+---------------------------------------------
+
+=== Generating documentation
+
+This builder is the one which is the least clean of all. It uses the Debian
+packaging information to decide which docs to publish and which manpages to
+generate. Additionally, it uses a for loop instead of calling a script. I
+recommend including a script to do this in your repository instead.
+
+Apart from these concerns, the builder is straight-forward: It clones the git
+repository, generates the documentation and then uploads the documentation to
+the buildmaster:
+
+*Generating documentation*:
+---------------------------------------------
+f = factories['docs'] = BuildFactory()
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
+
+f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'COPY-DOCS',
+ masterdest = 'htdocs/docs-git',
+ compress = 'bz2',
+ name = 'upload docs'))
+
+f.addStep(DocsToIRC())
+---------------------------------------------
+
+Just as +ClangToIRC+, +DocsToIRC+ appends a static message:
+
+*DocsToIRC*:
+---------------------------------------------
+class DocsToIRC(buildstep.BuildStep):
+ def start(self):
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
+ self.finished(SUCCESS)
+---------------------------------------------
+
+=== Building Debian/Ubuntu packages
+
+This is the most complex builder of all. It uses +pbuilder-dist+, +debchange+,
++dpkg-buildpackage+ and +reprepro+ to generate a Debian repository with a
+cleanly compiled package for amd64 and i386. In order for it to work, you need
+to install the following packages: +apt-get install devscripts dpkg-dev
+reprepro ubuntu-dev-tools+. Afterwards, you need to allow the user as which the
+buildslave runs to execute pbuilder via sudo without needing a password, so run
++visudo+ and add a line like this one:
+
+*sudoers line*:
+---------------------------------------------
+build ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder
+---------------------------------------------
+
+Then, as the user as which your buildslave runs, setup the pbuilder
+environments (you only need to do this once):
+
+*pbuilder preparation*:
+---------------------------------------------
+sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64
+sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386
+pbuilder-sid-amd64 create
+pbuilder-sid-i386 create
+---------------------------------------------
+
+Also, you will need a GPG key to sign these packages.
+
+The debian builder starts by unpacking the dist tarball, copying the Debian
+packaging from git, creating an empty Debian repository with the
++i3-autobuild-keyring+ contents in it. It then adds a new changelog entry to
+reflect the git version and the fact that this package was built automatically,
+builds a source package with +dpkg-buildpackage+ and adds it to the repository.
+Afterwards, it updates each pbuilder and builds binary packages for each
+architecture (amd64 and i386). After adding the resulting packages to the
+repository, it uploads the repository to the buildmaster:
+
+*Debian builder*:
+---------------------------------------------
+distributions = [ 'sid-amd64', 'sid-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['debian-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name = 'copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+ name = 'update changelog',
+ workdir = 'build/DIST',
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+ name = 'source pkg',
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+ workdir = 'build/DIST',
+)
+
+for dist in distributions:
+ f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+ """Codename: sid
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+ slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f,
+ name = 'clone keyring repo',
+ command = 'git clone git://code.i3wm.org/i3-autobuild-keyring',
+)
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+ # update the pbuilder
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+ # build the package for each dist
+ f.addStep(ShellCommand(
+ logEnviron = False,
+ name = 'pkg ' + dist,
+ command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+ warnOnFailure = True
+ ))
+
+ reprepro_include(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+ f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'REPO-sid/' + directory,
+ masterdest = 'htdocs/debian/sid/' + directory,
+ compress = 'bz2',
+ name = 'upload sid ' + directory,
+ haltOnFailure = True,
+ ))
+
+f.addStep(master.MasterShellCommand(
+ command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
+ name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+ command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
+ name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+ command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
+ name = 'touch i18n/Index',
+))
+---------------------------------------------
+
+The +reprepro_include+ command is defined as follows:
+
+*reprepro_include*:
+---------------------------------------------
+def reprepro_include(factory, path, debtype='deb', **kwargs):
+ cmd(factory,
+ name = 'reprepro include',
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
+ **kwargs
+ )
+---------------------------------------------
+
+Running such a builder for Ubuntu works exactly the same way, but you need to
+replace "sid" with "precise" in all places (see the full configuration file for
+an example).
+
+=== Status targets
+
+We don’t advertise the HTTP status target. Instead, status is posted to IRC via
+a custom bot. This bot provides an HTTP end point and buildbot is configured to
+push status changes to that endpoint:
+
+*http status target*:
+---------------------------------------------
+c['status'].append(buildbot.status.status_push.HttpStatusPush(
+ serverUrl = 'http://localhost:8080/push_buildbot',
+))
+---------------------------------------------
+
+You can find the source code of that bot at
+http://code.stapelberg.de/git/go-buildbot-announce/. As the name suggests, it
+is written in Go. Also, it is quite specific to i3, so you might be better off
+implementing such a bot (or plugin) on your own. It might make for a nice
+example, though, especially back when its only feature was announcing the build
+status:
+
+http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0
+
+== Full configuration file
+
+This is the full configuration file, as tested and currently in use (except for
+the passwords, though):
+
+*master.cfg*:
+---------------------------------------------
+# -*- python -*-
+# -*- coding: utf-8
+# vim:ts=4:sw=4:expandtab:syntax=python
+#
+# i3 buildbot configuration
+# © 2012 Michael Stapelberg, Public Domain
+# see http://i3wm.org/docs/buildbot.html for more information.
+
+from buildbot.buildslave import BuildSlave
+from buildbot.changes import pb
+from buildbot.schedulers.basic import SingleBranchScheduler
+from buildbot.schedulers.triggerable import Triggerable
+from buildbot.process.properties import WithProperties
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.source.git import Git
+from buildbot.steps.shell import ShellCommand
+from buildbot.steps.shell import Compile
+from buildbot.steps.trigger import Trigger
+from buildbot.steps import shell, transfer, master, slave
+from buildbot.config import BuilderConfig
+from buildbot.process import buildstep
+from buildbot.status import html
+from buildbot.status import words
+import buildbot.status.status_push
+from buildbot.status.web import auth, authz
+from buildbot.status.builder import SUCCESS, FAILURE
+
+c = BuildmasterConfig = {}
+
+c['slaves'] = [BuildSlave('docsteel-vm', 'secret')]
+c['slavePortnum'] = 9989
+# Changes are pushed to buildbot using a git hook.
+c['change_source'] = [pb.PBChangeSource(
+ user = 'i3-source',
+ passwd = 'secret',
+)]
+
+################################################################################
+# schedulers
+################################################################################
+
+c['schedulers'] = []
+
+# The first scheduler kicks off multiple builders:
+# • 'dist' builds a dist tarball and starts the triggerable schedulers
+# 'compile'
+# • 'docs' builds the documentation with a special asciidoc configuration
+# (therefore, it does not profit from a dist tarball and can be run in
+# parallel).
+c['schedulers'].append(SingleBranchScheduler(
+ name = 'dist',
+ branch = 'next',
+ treeStableTimer = 10,
+ builderNames = [ 'dist', 'docs' ],
+))
+
+c['schedulers'].append(Triggerable(
+ name = 'dist-tarball-done',
+ builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
+))
+
+################################################################################
+# Shortcuts for builders
+################################################################################
+
+# shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False
+def cmd(factory, **kwargs):
+ factory.addStep(ShellCommand(
+ haltOnFailure=True,
+ logEnviron=False,
+ **kwargs
+ ))
+
+# Shortcut to add steps necessary to download and unpack the dist tarball.
+def unpack_dist_tarball(factory):
+ factory.addStep(transfer.FileDownload(
+ mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+ slavedest='dist.tar.bz2',
+ ))
+ factory.addStep(slave.MakeDirectory(dir='build/DIST'))
+ cmd(factory,
+ name = 'unpack dist tarball',
+ command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
+ )
+
+# Includes the given path in REPO-sid using reprepro.
+def reprepro_include(factory, path, debtype='deb', **kwargs):
+ cmd(factory,
+ name = 'reprepro include',
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
+ **kwargs
+ )
+
+def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs):
+ cmd(factory,
+ name = 'reprepro include',
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path,
+ **kwargs
+ )
+
+################################################################################
+# Custom steps
+################################################################################
+
+# Adds the ircsuffix property to reflect whether there were warnings.
+class WarningsToIRC(buildstep.BuildStep):
+ def start(self):
+ warnings = self.getProperty("warnings-count")
+ if warnings is not None and int(warnings) > 0:
+ warnings = int(warnings) # just to be sure
+ self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else ""))
+ else:
+ self.setProperty("ircsuffix", "\0033 without warnings")
+ self.finished(SUCCESS)
+
+# Adds a link to the automatically generated documentation.
+class DocsToIRC(buildstep.BuildStep):
+ def start(self):
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
+ self.finished(SUCCESS)
+
+# Adds a link to the clang report.
+class ClangToIRC(buildstep.BuildStep):
+ def start(self):
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
+ self.finished(SUCCESS)
+
+################################################################################
+# Shared steps, used in different factories.
+################################################################################
+
+s_git = Git(
+ repourl='git://code.i3wm.org/i3',
+ branch='next',
+
+ # Check out the latest revision, not the one which caused this build.
+ alwaysUseLatest=True,
+
+ # We cannot use shallow because it breaks git describe --tags.
+ shallow=False,
+
+ # Delete remnants of previous builds.
+ mode='full',
+
+ # Store checkouts in source/ and copy them over to build/ to save
+ # bandwidth.
+ method='copy',
+
+ # XXX: In newer versions of buildbot (> 0.8.6), we want to use
+ # getDescription={ 'tags': True } here and get rid of the extra git
+ # describe --tags step.
+)
+
+################################################################################
+# factory: "dist" — builds the dist tarball once (used by all other factories)
+################################################################################
+
+factories = {}
+
+f = factories['dist'] = BuildFactory()
+# Check out the git repository.
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+# Build the dist tarball.
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
+# Rename the created tarball to a well-known name.
+cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'))
+# Upload the dist tarball to the master (other factories download it later).
+f.addStep(transfer.FileUpload(
+ slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
+ masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+))
+# Cleanup old dist tarballs (everything older than tree days).
+f.addStep(master.MasterShellCommand(
+ command = "find distballs -mtime +3 -exec rm '{}' \;",
+ name = 'cleanup old dist tarballs',
+))
+# Everything worked fine, now trigger compilation.
+f.addStep(Trigger(
+ schedulerNames = [ 'dist-tarball-done' ],
+ copy_properties = [ 'gitversion' ],
+))
+
+################################################################################
+# factory: "compile" — compiles the dist tarball and reports warnings
+################################################################################
+
+f = factories['compile'] = BuildFactory()
+unpack_dist_tarball(f)
+f.addStep(Compile(
+ command = [ 'make', 'DEBUG=0', '-j4' ],
+ warningPattern = '.*warning: ',
+ warnOnWarnings = True,
+ workdir = 'build/DIST',
+ env = {
+ 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
+ 'CFLAGS': '-Wformat -Wformat-security'
+ },
+))
+
+f.addStep(WarningsToIRC())
+
+################################################################################
+# factory: "clang-analyze" — runs a static code analysis
+################################################################################
+# $ sudo apt-get install clang
+
+f = factories['clang-analyze'] = BuildFactory()
+unpack_dist_tarball(f)
+cmd(f,
+ name='analyze',
+ command = [
+ 'scan-build',
+ '-o', '../CLANG',
+ '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
+ 'make', '-j8',
+ ],
+ workdir = 'build/DIST',
+)
+
+# remove the subdirectory -- we always want to overwrite
+cmd(f, command = 'mv CLANG/*/* CLANG/')
+
+f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'CLANG',
+ masterdest = 'htdocs/clang-analyze',
+ compress = 'bz2',
+ name = 'upload output',
+))
+
+f.addStep(ClangToIRC())
+
+################################################################################
+# factory: "docs" — builds documentation with a special asciidoc conf
+################################################################################
+
+f = factories['docs'] = BuildFactory()
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
+
+f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'COPY-DOCS',
+ masterdest = 'htdocs/docs-git',
+ compress = 'bz2',
+ name = 'upload docs'))
+
+f.addStep(DocsToIRC())
+
+################################################################################
+# factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386
+################################################################################
+
+distributions = [ 'sid-amd64', 'sid-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['debian-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+ name = 'update changelog',
+ workdir = 'build/DIST',
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+ name = 'source pkg',
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+ workdir = 'build/DIST',
+)
+
+for dist in distributions:
+ f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+ """Codename: sid
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+ slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+ # update the pbuilder
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+ # build the package for each dist
+ f.addStep(ShellCommand(
+ logEnviron = False,
+ name = 'pkg ' + dist,
+ command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+ warnOnFailure = True
+ ))
+
+ reprepro_include(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+ f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'REPO-sid/' + directory,
+ masterdest = 'htdocs/debian/sid/' + directory,
+ compress = 'bz2',
+ name = 'upload sid ' + directory,
+ haltOnFailure = True,
+ ))
+
+f.addStep(master.MasterShellCommand(
+ command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
+ name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+ command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
+ name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+ command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
+ name = 'touch i18n/Index',
+))
+
+################################################################################
+# factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386
+################################################################################
+
+distributions = [ 'precise-amd64', 'precise-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['ubuntu-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+ name = 'update changelog',
+ workdir = 'build/DIST',
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+ name = 'source pkg',
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+ workdir = 'build/DIST',
+)
+
+for dist in distributions:
+ f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+ """Codename: precise
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+ slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
+reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+ # update the pbuilder
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+ # build the package for each dist
+ f.addStep(ShellCommand(
+ logEnviron = False,
+ name = 'pkg ' + dist,
+ command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+ warnOnFailure = True
+ ))
+
+ reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+ f.addStep(transfer.DirectoryUpload(
+ slavesrc = 'REPO-sid/' + directory,
+ masterdest = 'htdocs/ubuntu/precise/' + directory,
+ compress = 'bz2',
+ name = 'upload precise ' + directory,
+ haltOnFailure = True,
+ ))
+
+f.addStep(master.MasterShellCommand(
+ command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;",
+ name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+ command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ],
+ name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+ command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ],
+ name = 'touch i18n/Index',
+))
+
+
+c['builders'] = []
+
+# Add all builders to all buildslaves.
+for factoryname in factories.keys():
+ c['builders'].append(BuilderConfig(
+ name = factoryname,
+ slavenames=['docsteel-vm'],
+ factory=factories[factoryname],
+ ))
+
+
+####### STATUS TARGETS
+
+c['status'] = []
+
+authz_cfg=authz.Authz(
+ gracefulShutdown = False,
+ forceBuild = False,
+ forceAllBuilds = False,
+ pingBuilder = False,
+ stopBuild = False,
+ stopAllBuilds = False,
+ cancelPendingBuild = False,
+)
+
+c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
+
+c['status'].append(buildbot.status.status_push.HttpStatusPush(
+ serverUrl = 'http://localhost:8080/push_buildbot',
+))
+
+####### PROJECT IDENTITY
+
+c['title'] = 'i3'
+c['titleURL'] = 'http://i3wm.org/'
+# Removed so that search engines don’t crawl it
+c['buildbotURL'] = 'http://localhost/'
+
+####### DB URL
+
+c['db'] = {
+ # This specifies what database buildbot uses to store its state. You can leave
+ # this at its default for all but the largest installations.
+ 'db_url' : "sqlite:///state.sqlite",
+}
+---------------------------------------------
--- /dev/null
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"\r
+ "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">\r
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">\r
+<head>\r
+<link rel="icon" type="image/png" href="/favicon.png">\r
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />\r
+<meta name="generator" content="AsciiDoc 8.6.7" />\r
+<title>i3: The i3 buildbot setup</title>\r
+<link rel="stylesheet" href="/css/style.css" type="text/css" />\r
+<link rel="stylesheet" href="/css/xhtml11.css" type="text/css" />\r
+<script type="text/javascript">\r
+/*<![CDATA[*/\r
+window.onload = function(){asciidoc.footnotes(); asciidoc.toc(2);}\r
+/*]]>*/\r
+</script>\r
+<script type="text/javascript" src="/js/asciidoc-xhtml11.js"></script>\r
+</head>\r
+<body class="article">\r
+\r
+ <div id="main">\r
+ <a href="/"><h1 id="title">i3 - improved tiling WM</h1></a>\r
+ <ul id="nav">\r
+ <li><a style="border-bottom: 2px solid #fff" href="/docs">Docs</a></li>\r
+ <li><a href="/screenshots">Screens</a></li>\r
+ <li><a href="/contact">Contact</a></li>\r
+ <li><a href="http://bugs.i3wm.org/">Bugs</a></li>\r
+ </ul>\r
+ <br style="clear: both">\r
+<div id="content">\r
+<div id="header">\r
+<h1>The i3 buildbot setup</h1>\r
+<span id="author">Michael Stapelberg</span><br />\r
+<span id="email"><tt><<a href="mailto:michael@i3wm.org">michael@i3wm.org</a>></tt></span><br />\r
+<span id="revdate">August 2012</span>\r
+<div id="toc">
+ <div id="toctitle">Table of Contents</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+</div>\r
+</div>\r
+<div id="preamble">\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>This document explains the <a href="http://www.buildbot.net/">buildbot</a> setup we use to\r
+provide up-to-date documentation and debian packages at <a href="http://build.i3wm.org/">http://build.i3wm.org/</a>.\r
+We publish these information so that our setup is well-documented (thus\r
+decreasing future maintenance effort) and because it might be interesting for\r
+other projects.</p></div>\r
+</div>\r
+</div>\r
+<div class="sect1">\r
+<h2 id="_introduction">1. Introduction</h2>\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>What we are doing in i3 is called Continuous Integration (see\r
+<a href="http://en.wikipedia.org/wiki/Continuous_integration">http://en.wikipedia.org/wiki/Continuous_integration</a>): we publish the changes we\r
+make on our local machines as often as possible. In order to maintain a\r
+continuously high quality, each time any developer pushes changes to the\r
+official git repository, a number of quality assurance tools start running\r
+automatically:</p></div>\r
+<div class="olist arabic"><ol class="arabic">\r
+<li>\r
+<p>\r
+Latest documentation is generated and provided at\r
+ <a href="http://build.i3wm.org/docs/">http://build.i3wm.org/docs/</a>. This makes it easy to link to documentation for\r
+ features which are only in the current git version, not in the released\r
+ version.\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+The source code is compiled and it is automatically posted to the IRC\r
+ channel whether there were any compiler warnings. While developers should\r
+ notice compiler warnings, this mechanism creates a bit of public pressure\r
+ ("Oh, Michael introduced warnings with this commit!"). More importantly,\r
+ since this mechanism builds a dist tarball and then compiles that tarball,\r
+ any changes to the source which would result in an uncompilable dist tarball\r
+ are immediately obvious. Therefore, we could cut a release from the current\r
+ git version at any point in time.\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+The clang static analyzer runs and the resulting report is provided at\r
+ <a href="http://build.i3wm.org/clang-analyze/">http://build.i3wm.org/clang-analyze/</a>. While every developer needs to compile\r
+ his code before committing, he doesn’t necessarily use clang (so we catch\r
+ build failures when using clang) and he also probably doesn’t run a static\r
+ analyzer as part of his normal workflow. By just being available without any\r
+ friction, this mechanism encourages developers to look at the report and fix\r
+ problems.\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+Debian (and Ubuntu) packages are built. This not only ensures that we don’t\r
+ change anything in the source code which would lead to an FTBFS (Fails To\r
+ Build From Source) when building a Debian package, it also goes a long way\r
+ to encourage end users to test i3. To remove the need and resource\r
+ requirements for them to compile their own version of i3 regularly, we\r
+ provide packages that integrate conveniently with a normal Debian system\r
+ (e.g. that are automatically upgraded).\r
+</p>\r
+</li>\r
+</ol></div>\r
+</div>\r
+</div>\r
+<div class="sect1">\r
+<h2 id="_why_buildbot">2. Why buildbot?</h2>\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>Previously, I was unsatisfied with the current state of FOSS CI tools like\r
+Jenkins, Tinderbox and others. They either seemed bloated, hard to use,\r
+outdated or unappealing for some other reason.</p></div>\r
+<div class="paragraph"><p>Then I discovered buildbot and was impressed by its flexibility. It let me\r
+implement everything I wanted from a CI tool and (in my opinion) it is\r
+light-weight, easy to deploy and well maintained.</p></div>\r
+<div class="paragraph"><p>The only downside of buildbot is its configuration and documentation: You need\r
+to spend quite a bit of time (I needed multiple days) until it works the way\r
+you want it to and oftentimes, the documentation is far too sparse. This is one\r
+of the reasons why I’m publishing the i3 setup.</p></div>\r
+</div>\r
+</div>\r
+<div class="sect1">\r
+<h2 id="_configuration">3. Configuration</h2>\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>See the next section for a complete, copy & pasteable configuration file. This\r
+section covers the most important aspects without covering every line.</p></div>\r
+<div class="paragraph"><p>This document assumes you are running buildbot 0.8.6p1.</p></div>\r
+<div class="sect2">\r
+<h3 id="_change_sources">3.1. Change sources</h3>\r
+<div class="paragraph"><p>Since i3 uses a central git repository, we use the official buildbot\r
+<a href="https://github.com/buildbot/buildbot/blob/master/master/contrib/git_buildbot.py">git\r
+post-receive hook</a> that sends the change information to the buildbot master.</p></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_schedulers">3.2. Schedulers</h3>\r
+<div class="paragraph"><p>There are two things (called "builders" in buildbot-language) which happen\r
+whenever a new change in the <tt>next</tt> branch of i3 occurs:</p></div>\r
+<div class="olist arabic"><ol class="arabic">\r
+<li>\r
+<p>\r
+The "docs" builder builds and uploads the latest documentation. This happens\r
+ directly from the git repository with a custom asciidoc configuration which\r
+ indicates that these docs refer to the git version. Therefore, this builder\r
+ does not benefit from having a dist tarball available (contrary to the other\r
+ builders).\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+The "dist" builder prepares a dist tarball and then triggers the remaining\r
+ builders. This ensures that building the dist tarball (an operation which\r
+ takes about one minute due to documentation generation) only happens once.\r
+</p>\r
+</li>\r
+</ol></div>\r
+<div class="paragraph"><p>Here is the relevant configuration part:</p></div>\r
+<div class="paragraph"><p><strong>Schedulers</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>c['schedulers'] = []\r
+\r
+c['schedulers'].append(SingleBranchScheduler(\r
+ name = 'dist',\r
+ branch = 'next',\r
+ treeStableTimer = 10,\r
+ builderNames = [ 'dist', 'docs' ],\r
+))\r
+\r
+c['schedulers'].append(Triggerable(\r
+ name = 'dist-tarball-done',\r
+ builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],\r
+))</tt></pre>\r
+</div></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_building_the_dist_tarball">3.3. Building the dist tarball</h3>\r
+<div class="paragraph"><p>This builder clones the i3 git repository and runs "make dist", which creates a\r
+tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then\r
+renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable\r
+name in the next steps) and uploaded to the buildbot master (since we can have\r
+multiple buildslaves, we cannot just let it rest on the buildslave that built\r
+it). Afterwards, old dist tarballs are cleaned up and the remaining builders\r
+are triggered:</p></div>\r
+<div class="paragraph"><p><strong>Building a dist tarball</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>factories = {}\r
+\r
+f = factories['dist'] = BuildFactory()\r
+\r
+# Check out the git repository.\r
+f.addStep(s_git)\r
+\r
+# Fill the 'gitversion' property with the output of git describe --tags.\r
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))\r
+\r
+# Build the dist tarball.\r
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])\r
+\r
+# Rename the created tarball to a well-known name.\r
+cmd(f,\r
+ name = 'rename tarball',\r
+ command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'),\r
+)\r
+\r
+# Upload the dist tarball to the master (other factories download it later).\r
+f.addStep(transfer.FileUpload(\r
+ slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),\r
+ masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),\r
+))\r
+\r
+# Cleanup old dist tarballs (everything older than tree days).\r
+f.addStep(master.MasterShellCommand(\r
+ command = "find distballs -mtime +3 -exec rm '{}' \;",\r
+ name = 'cleanup old dist tarballs',\r
+))\r
+\r
+# Everything worked fine, now trigger compilation.\r
+f.addStep(Trigger(\r
+ schedulerNames = [ 'dist-tarball-done' ],\r
+ copy_properties = [ 'gitversion' ],\r
+))</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Three things are noteworthy about this part of the configuration:</p></div>\r
+<div class="olist arabic"><ol class="arabic">\r
+<li>\r
+<p>\r
+For convenience, we call each factory <tt>f</tt> (just like the global buildbot\r
+ config uses <tt>c</tt> for the top-level configuration) and add it to a dictionary.\r
+ Factories in that dictionary are later automatically configured for each\r
+ buildslave.\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+We have a shared step called <tt>s_git</tt> so that we only have one location in\r
+ the configuration file where we specify the git repository URL and branch.\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+We have a custom function called <tt>cmd</tt> which is a shortcut for defining a\r
+ <tt>ShellCommand</tt> with <tt>haltOnFailure=True</tt> (since each step is critical) and\r
+ <tt>logEnviron=False</tt> (for brevity).\r
+</p>\r
+</li>\r
+</ol></div>\r
+<div class="paragraph"><p>Here are their definitions:</p></div>\r
+<div class="paragraph"><p><strong>cmd</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>def cmd(factory, **kwargs):\r
+ factory.addStep(ShellCommand(\r
+ haltOnFailure = True,\r
+ logEnviron = False,\r
+ **kwargs\r
+ ))</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p><strong>s_git</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>s_git = Git(\r
+ repourl = 'git://code.i3wm.org/i3',\r
+ branch = 'next',\r
+\r
+ # Check out the latest revision, not the one which caused this build.\r
+ alwaysUseLatest = True,\r
+\r
+ # We cannot use shallow because it breaks git describe --tags.\r
+ shallow = False,\r
+\r
+ # Delete remnants of previous builds.\r
+ mode = 'full',\r
+\r
+ # Store checkouts in source/ and copy them over to build/ to save\r
+ # bandwidth.\r
+ method = 'copy',\r
+)</tt></pre>\r
+</div></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_compiling_the_dist_tarball">3.4. Compiling the dist tarball</h3>\r
+<div class="paragraph"><p>For this builder to work, you obviously need to install all the\r
+build-dependencies for your software on each buildslave. In the case of i3,\r
+this can be done with <tt>apt-get build-dep i3-wm</tt>.</p></div>\r
+<div class="paragraph"><p>The compilation is pretty straight-forward since it uses the builtin <tt>Compile</tt>\r
+step. We call <tt>make</tt> with <tt>-j4</tt> (we don’t have enough buildslaves to make\r
+figuring out the amount of cores at build-time worthwhile) and <tt>DEBUG=0</tt> to\r
+simulate release build conditions. Also, we pass the preprocessor flag\r
+<tt>-D_FORTIFY_SOURCE=2</tt> and the compiler flags <tt>-Wformat</tt> and <tt>-Wformat-security</tt>\r
+to enable additional warnings.</p></div>\r
+<div class="paragraph"><p><strong>Compiling the dist tarball</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>f = factories['compile'] = BuildFactory()\r
+unpack_dist_tarball(f)\r
+f.addStep(Compile(\r
+ command = [ 'make', 'DEBUG=0', '-j4' ],\r
+ warningPattern = '.*warning: ',\r
+ warnOnWarnings = True,\r
+ workdir = 'build/DIST',\r
+ env = {\r
+ 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',\r
+ 'CFLAGS': '-Wformat -Wformat-security'\r
+ },\r
+))\r
+\r
+f.addStep(WarningsToIRC())</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Again, we use custom functions (and a custom buildstep) to make our lives\r
+easier. Here is the definition of unpack_dist_tarball which adds three steps to\r
+the factory that download and unpack the dist tarball to the <tt>DIST/</tt> directory:</p></div>\r
+<div class="paragraph"><p><strong>unpack_dist_tarball</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>def unpack_dist_tarball(factory):\r
+ factory.addStep(transfer.FileDownload(\r
+ mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),\r
+ slavedest = 'dist.tar.bz2',\r
+ ))\r
+\r
+ factory.addStep(slave.MakeDirectory(dir = 'build/DIST'))\r
+\r
+ cmd(factory,\r
+ name = 'unpack dist tarball',\r
+ command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],\r
+ )</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>The <tt>WarningsToIRC</tt> build step is a custom build step which sets a property\r
+called "ircsuffix" that is used by our custom IRC bot. This is covered later in\r
+more detail. This property gets set to a green or red message, depending on\r
+whether there were any warnings:</p></div>\r
+<div class="paragraph"><p><strong>WarningsToIRC</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>class WarningsToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ warnings = self.getProperty("warnings-count")\r
+ if warnings is not None and int(warnings) > 0:\r
+ warnings = int(warnings) # just to be sure\r
+ self.setProperty("ircsuffix", ("\0037 with %d warning%s!" %\r
+ (warnings, "s" if warnings != 1 else "")))\r
+ else:\r
+ self.setProperty("ircsuffix", "\0033 without warnings")\r
+ self.finished(SUCCESS)</tt></pre>\r
+</div></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_static_code_analysis">3.5. Static code analysis</h3>\r
+<div class="paragraph"><p>For this builder to work, you additionally need the <tt>clang</tt> compiler on each\r
+buildslave: <tt>apt-get install clang</tt>.</p></div>\r
+<div class="paragraph"><p>This builder uses only custom functions which you already know by now. It runs\r
+scan-build, then moves scan-build’s output from a date-based directory directly\r
+into the <tt>CLANG/</tt> directory and uploads that to the buildmaster.</p></div>\r
+<div class="paragraph"><p>On the buildmaster, a webserver is configured which has a symlink to\r
+<tt>/home/build/i3-master/htdocs/clang-analyze</tt> in its document root.</p></div>\r
+<div class="paragraph"><p><strong>static code analysis</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>f = factories['clang-analyze'] = BuildFactory()\r
+unpack_dist_tarball(f)\r
+cmd(f,\r
+ name='analyze',\r
+ command = [\r
+ 'scan-build',\r
+ '-o', '../CLANG',\r
+ '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),\r
+ 'make', '-j8',\r
+ ],\r
+ workdir = 'build/DIST',\r
+)\r
+\r
+# remove the subdirectory -- we always want to overwrite\r
+cmd(f, command = 'mv CLANG/*/* CLANG/')\r
+\r
+f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'CLANG',\r
+ masterdest = 'htdocs/clang-analyze',\r
+ compress = 'bz2',\r
+ name = 'upload output',\r
+))\r
+\r
+f.addStep(ClangToIRC())</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>The <tt>ClangToIRC</tt> custom step is even simpler than <tt>WarningsToIRC</tt>. It simply\r
+sets the ircsuffix property to a static message:</p></div>\r
+<div class="paragraph"><p><strong>ClangToIRC</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>class ClangToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")\r
+ self.finished(SUCCESS)</tt></pre>\r
+</div></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_generating_documentation">3.6. Generating documentation</h3>\r
+<div class="paragraph"><p>This builder is the one which is the least clean of all. It uses the Debian\r
+packaging information to decide which docs to publish and which manpages to\r
+generate. Additionally, it uses a for loop instead of calling a script. I\r
+recommend including a script to do this in your repository instead.</p></div>\r
+<div class="paragraph"><p>Apart from these concerns, the builder is straight-forward: It clones the git\r
+repository, generates the documentation and then uploads the documentation to\r
+the buildmaster:</p></div>\r
+<div class="paragraph"><p><strong>Generating documentation</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>f = factories['docs'] = BuildFactory()\r
+f.addStep(s_git)\r
+# Fill the 'gitversion' property with the output of git describe --tags.\r
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))\r
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])\r
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")\r
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))\r
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")\r
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")\r
+\r
+f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'COPY-DOCS',\r
+ masterdest = 'htdocs/docs-git',\r
+ compress = 'bz2',\r
+ name = 'upload docs'))\r
+\r
+f.addStep(DocsToIRC())</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Just as <tt>ClangToIRC</tt>, <tt>DocsToIRC</tt> appends a static message:</p></div>\r
+<div class="paragraph"><p><strong>DocsToIRC</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>class DocsToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")\r
+ self.finished(SUCCESS)</tt></pre>\r
+</div></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_building_debian_ubuntu_packages">3.7. Building Debian/Ubuntu packages</h3>\r
+<div class="paragraph"><p>This is the most complex builder of all. It uses <tt>pbuilder-dist</tt>, <tt>debchange</tt>,\r
+<tt>dpkg-buildpackage</tt> and <tt>reprepro</tt> to generate a Debian repository with a\r
+cleanly compiled package for amd64 and i386. In order for it to work, you need\r
+to install the following packages: <tt>apt-get install devscripts dpkg-dev\r
+reprepro ubuntu-dev-tools</tt>. Afterwards, you need to allow the user as which the\r
+buildslave runs to execute pbuilder via sudo without needing a password, so run\r
+<tt>visudo</tt> and add a line like this one:</p></div>\r
+<div class="paragraph"><p><strong>sudoers line</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>build ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Then, as the user as which your buildslave runs, setup the pbuilder\r
+environments (you only need to do this once):</p></div>\r
+<div class="paragraph"><p><strong>pbuilder preparation</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64\r
+sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386\r
+pbuilder-sid-amd64 create\r
+pbuilder-sid-i386 create</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Also, you will need a GPG key to sign these packages.</p></div>\r
+<div class="paragraph"><p>The debian builder starts by unpacking the dist tarball, copying the Debian\r
+packaging from git, creating an empty Debian repository with the\r
+<tt>i3-autobuild-keyring</tt> contents in it. It then adds a new changelog entry to\r
+reflect the git version and the fact that this package was built automatically,\r
+builds a source package with <tt>dpkg-buildpackage</tt> and adds it to the repository.\r
+Afterwards, it updates each pbuilder and builds binary packages for each\r
+architecture (amd64 and i386). After adding the resulting packages to the\r
+repository, it uploads the repository to the buildmaster:</p></div>\r
+<div class="paragraph"><p><strong>Debian builder</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>distributions = [ 'sid-amd64', 'sid-i386' ]\r
+gpg_key = 'BE1DB1F1'\r
+\r
+f = factories['debian-packages'] = BuildFactory()\r
+# We need the git repository for the Debian packaging.\r
+f.addStep(s_git)\r
+unpack_dist_tarball(f)\r
+cmd(f, name = 'copy packaging', command = "cp -r debian DIST/")\r
+\r
+# Add a new changelog entry to have the git version in the package version.\r
+cmd(f,\r
+ name = 'update changelog',\r
+ workdir = 'build/DIST',\r
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],\r
+)\r
+\r
+cmd(f,\r
+ name = 'source pkg',\r
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],\r
+ workdir = 'build/DIST',\r
+)\r
+\r
+for dist in distributions:\r
+ f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist))\r
+\r
+# Create debian sid repository\r
+f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf'))\r
+f.addStep(transfer.StringDownload(\r
+ """Codename: sid\r
+Suite: unstable\r
+Architectures: i386 amd64 source\r
+Components: main\r
+DebIndices: Packages Release . .gz .bz2\r
+DscIndices: Sources Release . .gz .bz2\r
+SignWith: %(gpg_key)s\r
+""" % { "gpg_key": gpg_key },\r
+ slavedest = 'REPO-sid/conf/distributions',\r
+))\r
+\r
+# add source package to repository\r
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')\r
+\r
+# Add keyring to the repository. We need to run git clone on our own because\r
+# the Git() step assumes there’s precisely one repository we want to deal with.\r
+# No big deal since the i3-autobuild-keyring repository is not big.\r
+cmd(f,\r
+ name = 'clone keyring repo',\r
+ command = 'git clone git://code.i3wm.org/i3-autobuild-keyring',\r
+)\r
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')\r
+\r
+for dist in distributions:\r
+ # update the pbuilder\r
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')\r
+\r
+ # build the package for each dist\r
+ f.addStep(ShellCommand(\r
+ logEnviron = False,\r
+ name = 'pkg ' + dist,\r
+ command = 'pbuilder-' + dist + ' build --binary-arch \\r
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',\r
+ warnOnFailure = True\r
+ ))\r
+\r
+ reprepro_include(f, 'RESULT-' + dist + '/*.changes')\r
+\r
+# upload the sid repo\r
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we\r
+# prefer providing old packages over providing no packages at all :).\r
+for directory in [ 'pool', 'dists' ]:\r
+ f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'REPO-sid/' + directory,\r
+ masterdest = 'htdocs/debian/sid/' + directory,\r
+ compress = 'bz2',\r
+ name = 'upload sid ' + directory,\r
+ haltOnFailure = True,\r
+ ))\r
+\r
+f.addStep(master.MasterShellCommand(\r
+ command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",\r
+ name = 'cleanup old packages',\r
+))\r
+\r
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not\r
+# try to download Translation-*)\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],\r
+ name = 'create i18n folder',\r
+))\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],\r
+ name = 'touch i18n/Index',\r
+))</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>The <tt>reprepro_include</tt> command is defined as follows:</p></div>\r
+<div class="paragraph"><p><strong>reprepro_include</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>def reprepro_include(factory, path, debtype='deb', **kwargs):\r
+ cmd(factory,\r
+ name = 'reprepro include',\r
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,\r
+ **kwargs\r
+ )</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Running such a builder for Ubuntu works exactly the same way, but you need to\r
+replace "sid" with "precise" in all places (see the full configuration file for\r
+an example).</p></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_status_targets">3.8. Status targets</h3>\r
+<div class="paragraph"><p>We don’t advertise the HTTP status target. Instead, status is posted to IRC via\r
+a custom bot. This bot provides an HTTP end point and buildbot is configured to\r
+push status changes to that endpoint:</p></div>\r
+<div class="paragraph"><p><strong>http status target</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>c['status'].append(buildbot.status.status_push.HttpStatusPush(\r
+ serverUrl = 'http://localhost:8080/push_buildbot',\r
+))</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>You can find the source code of that bot at\r
+<a href="http://code.stapelberg.de/git/go-buildbot-announce/">http://code.stapelberg.de/git/go-buildbot-announce/</a>. As the name suggests, it\r
+is written in Go. Also, it is quite specific to i3, so you might be better off\r
+implementing such a bot (or plugin) on your own. It might make for a nice\r
+example, though, especially back when its only feature was announcing the build\r
+status:</p></div>\r
+<div class="paragraph"><p><a href="http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0">http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0</a></p></div>\r
+</div>\r
+</div>\r
+</div>\r
+<div class="sect1">\r
+<h2 id="_full_configuration_file">4. Full configuration file</h2>\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>This is the full configuration file, as tested and currently in use (except for\r
+the passwords, though):</p></div>\r
+<div class="paragraph"><p><strong>master.cfg</strong>:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt># -*- python -*-\r
+# -*- coding: utf-8\r
+# vim:ts=4:sw=4:expandtab:syntax=python\r
+#\r
+# i3 buildbot configuration\r
+# © 2012 Michael Stapelberg, Public Domain\r
+# see http://i3wm.org/docs/buildbot.html for more information.\r
+\r
+from buildbot.buildslave import BuildSlave\r
+from buildbot.changes import pb\r
+from buildbot.schedulers.basic import SingleBranchScheduler\r
+from buildbot.schedulers.triggerable import Triggerable\r
+from buildbot.process.properties import WithProperties\r
+from buildbot.process.factory import BuildFactory\r
+from buildbot.steps.source.git import Git\r
+from buildbot.steps.shell import ShellCommand\r
+from buildbot.steps.shell import Compile\r
+from buildbot.steps.trigger import Trigger\r
+from buildbot.steps import shell, transfer, master, slave\r
+from buildbot.config import BuilderConfig\r
+from buildbot.process import buildstep\r
+from buildbot.status import html\r
+from buildbot.status import words\r
+import buildbot.status.status_push\r
+from buildbot.status.web import auth, authz\r
+from buildbot.status.builder import SUCCESS, FAILURE\r
+\r
+c = BuildmasterConfig = {}\r
+\r
+c['slaves'] = [BuildSlave('docsteel-vm', 'secret')]\r
+c['slavePortnum'] = 9989\r
+# Changes are pushed to buildbot using a git hook.\r
+c['change_source'] = [pb.PBChangeSource(\r
+ user = 'i3-source',\r
+ passwd = 'secret',\r
+)]\r
+\r
+################################################################################\r
+# schedulers\r
+################################################################################\r
+\r
+c['schedulers'] = []\r
+\r
+# The first scheduler kicks off multiple builders:\r
+# • 'dist' builds a dist tarball and starts the triggerable schedulers\r
+# 'compile'\r
+# • 'docs' builds the documentation with a special asciidoc configuration\r
+# (therefore, it does not profit from a dist tarball and can be run in\r
+# parallel).\r
+c['schedulers'].append(SingleBranchScheduler(\r
+ name = 'dist',\r
+ branch = 'next',\r
+ treeStableTimer = 10,\r
+ builderNames = [ 'dist', 'docs' ],\r
+))\r
+\r
+c['schedulers'].append(Triggerable(\r
+ name = 'dist-tarball-done',\r
+ builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],\r
+))\r
+\r
+################################################################################\r
+# Shortcuts for builders\r
+################################################################################\r
+\r
+# shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False\r
+def cmd(factory, **kwargs):\r
+ factory.addStep(ShellCommand(\r
+ haltOnFailure=True,\r
+ logEnviron=False,\r
+ **kwargs\r
+ ))\r
+\r
+# Shortcut to add steps necessary to download and unpack the dist tarball.\r
+def unpack_dist_tarball(factory):\r
+ factory.addStep(transfer.FileDownload(\r
+ mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),\r
+ slavedest='dist.tar.bz2',\r
+ ))\r
+ factory.addStep(slave.MakeDirectory(dir='build/DIST'))\r
+ cmd(factory,\r
+ name = 'unpack dist tarball',\r
+ command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],\r
+ )\r
+\r
+# Includes the given path in REPO-sid using reprepro.\r
+def reprepro_include(factory, path, debtype='deb', **kwargs):\r
+ cmd(factory,\r
+ name = 'reprepro include',\r
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,\r
+ **kwargs\r
+ )\r
+\r
+def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs):\r
+ cmd(factory,\r
+ name = 'reprepro include',\r
+ command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path,\r
+ **kwargs\r
+ )\r
+\r
+################################################################################\r
+# Custom steps\r
+################################################################################\r
+\r
+# Adds the ircsuffix property to reflect whether there were warnings.\r
+class WarningsToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ warnings = self.getProperty("warnings-count")\r
+ if warnings is not None and int(warnings) > 0:\r
+ warnings = int(warnings) # just to be sure\r
+ self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else ""))\r
+ else:\r
+ self.setProperty("ircsuffix", "\0033 without warnings")\r
+ self.finished(SUCCESS)\r
+\r
+# Adds a link to the automatically generated documentation.\r
+class DocsToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")\r
+ self.finished(SUCCESS)\r
+\r
+# Adds a link to the clang report.\r
+class ClangToIRC(buildstep.BuildStep):\r
+ def start(self):\r
+ self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")\r
+ self.finished(SUCCESS)\r
+\r
+################################################################################\r
+# Shared steps, used in different factories.\r
+################################################################################\r
+\r
+s_git = Git(\r
+ repourl='git://code.i3wm.org/i3',\r
+ branch='next',\r
+\r
+ # Check out the latest revision, not the one which caused this build.\r
+ alwaysUseLatest=True,\r
+\r
+ # We cannot use shallow because it breaks git describe --tags.\r
+ shallow=False,\r
+\r
+ # Delete remnants of previous builds.\r
+ mode='full',\r
+\r
+ # Store checkouts in source/ and copy them over to build/ to save\r
+ # bandwidth.\r
+ method='copy',\r
+\r
+ # XXX: In newer versions of buildbot (> 0.8.6), we want to use\r
+ # getDescription={ 'tags': True } here and get rid of the extra git\r
+ # describe --tags step.\r
+)\r
+\r
+################################################################################\r
+# factory: "dist" — builds the dist tarball once (used by all other factories)\r
+################################################################################\r
+\r
+factories = {}\r
+\r
+f = factories['dist'] = BuildFactory()\r
+# Check out the git repository.\r
+f.addStep(s_git)\r
+# Fill the 'gitversion' property with the output of git describe --tags.\r
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))\r
+# Build the dist tarball.\r
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])\r
+# Rename the created tarball to a well-known name.\r
+cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'))\r
+# Upload the dist tarball to the master (other factories download it later).\r
+f.addStep(transfer.FileUpload(\r
+ slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),\r
+ masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),\r
+))\r
+# Cleanup old dist tarballs (everything older than tree days).\r
+f.addStep(master.MasterShellCommand(\r
+ command = "find distballs -mtime +3 -exec rm '{}' \;",\r
+ name = 'cleanup old dist tarballs',\r
+))\r
+# Everything worked fine, now trigger compilation.\r
+f.addStep(Trigger(\r
+ schedulerNames = [ 'dist-tarball-done' ],\r
+ copy_properties = [ 'gitversion' ],\r
+))\r
+\r
+################################################################################\r
+# factory: "compile" — compiles the dist tarball and reports warnings\r
+################################################################################\r
+\r
+f = factories['compile'] = BuildFactory()\r
+unpack_dist_tarball(f)\r
+f.addStep(Compile(\r
+ command = [ 'make', 'DEBUG=0', '-j4' ],\r
+ warningPattern = '.*warning: ',\r
+ warnOnWarnings = True,\r
+ workdir = 'build/DIST',\r
+ env = {\r
+ 'CPPFLAGS': '-D_FORTIFY_SOURCE=2',\r
+ 'CFLAGS': '-Wformat -Wformat-security'\r
+ },\r
+))\r
+\r
+f.addStep(WarningsToIRC())\r
+\r
+################################################################################\r
+# factory: "clang-analyze" — runs a static code analysis\r
+################################################################################\r
+# $ sudo apt-get install clang\r
+\r
+f = factories['clang-analyze'] = BuildFactory()\r
+unpack_dist_tarball(f)\r
+cmd(f,\r
+ name='analyze',\r
+ command = [\r
+ 'scan-build',\r
+ '-o', '../CLANG',\r
+ '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),\r
+ 'make', '-j8',\r
+ ],\r
+ workdir = 'build/DIST',\r
+)\r
+\r
+# remove the subdirectory -- we always want to overwrite\r
+cmd(f, command = 'mv CLANG/*/* CLANG/')\r
+\r
+f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'CLANG',\r
+ masterdest = 'htdocs/clang-analyze',\r
+ compress = 'bz2',\r
+ name = 'upload output',\r
+))\r
+\r
+f.addStep(ClangToIRC())\r
+\r
+################################################################################\r
+# factory: "docs" — builds documentation with a special asciidoc conf\r
+################################################################################\r
+\r
+f = factories['docs'] = BuildFactory()\r
+f.addStep(s_git)\r
+# Fill the 'gitversion' property with the output of git describe --tags.\r
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))\r
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])\r
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")\r
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))\r
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")\r
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")\r
+\r
+f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'COPY-DOCS',\r
+ masterdest = 'htdocs/docs-git',\r
+ compress = 'bz2',\r
+ name = 'upload docs'))\r
+\r
+f.addStep(DocsToIRC())\r
+\r
+################################################################################\r
+# factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386\r
+################################################################################\r
+\r
+distributions = [ 'sid-amd64', 'sid-i386' ]\r
+gpg_key = 'BE1DB1F1'\r
+\r
+f = factories['debian-packages'] = BuildFactory()\r
+# We need the git repository for the Debian packaging.\r
+f.addStep(s_git)\r
+unpack_dist_tarball(f)\r
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")\r
+\r
+# Add a new changelog entry to have the git version in the package version.\r
+cmd(f,\r
+ name = 'update changelog',\r
+ workdir = 'build/DIST',\r
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],\r
+)\r
+\r
+cmd(f,\r
+ name = 'source pkg',\r
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],\r
+ workdir = 'build/DIST',\r
+)\r
+\r
+for dist in distributions:\r
+ f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))\r
+\r
+# Create debian sid repository\r
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))\r
+f.addStep(transfer.StringDownload(\r
+ """Codename: sid\r
+Suite: unstable\r
+Architectures: i386 amd64 source\r
+Components: main\r
+DebIndices: Packages Release . .gz .bz2\r
+DscIndices: Sources Release . .gz .bz2\r
+SignWith: %(gpg_key)s\r
+""" % { "gpg_key": gpg_key },\r
+ slavedest = 'REPO-sid/conf/distributions',\r
+))\r
+\r
+# add source package to repository\r
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')\r
+\r
+# Add keyring to the repository. We need to run git clone on our own because\r
+# the Git() step assumes there’s precisely one repository we want to deal with.\r
+# No big deal since the i3-autobuild-keyring repository is not big.\r
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')\r
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')\r
+\r
+for dist in distributions:\r
+ # update the pbuilder\r
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')\r
+\r
+ # build the package for each dist\r
+ f.addStep(ShellCommand(\r
+ logEnviron = False,\r
+ name = 'pkg ' + dist,\r
+ command = 'pbuilder-' + dist + ' build --binary-arch \\r
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',\r
+ warnOnFailure = True\r
+ ))\r
+\r
+ reprepro_include(f, 'RESULT-' + dist + '/*.changes')\r
+\r
+# upload the sid repo\r
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we\r
+# prefer providing old packages over providing no packages at all :).\r
+for directory in [ 'pool', 'dists' ]:\r
+ f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'REPO-sid/' + directory,\r
+ masterdest = 'htdocs/debian/sid/' + directory,\r
+ compress = 'bz2',\r
+ name = 'upload sid ' + directory,\r
+ haltOnFailure = True,\r
+ ))\r
+\r
+f.addStep(master.MasterShellCommand(\r
+ command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",\r
+ name = 'cleanup old packages',\r
+))\r
+\r
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not\r
+# try to download Translation-*)\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],\r
+ name = 'create i18n folder',\r
+))\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],\r
+ name = 'touch i18n/Index',\r
+))\r
+\r
+################################################################################\r
+# factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386\r
+################################################################################\r
+\r
+distributions = [ 'precise-amd64', 'precise-i386' ]\r
+gpg_key = 'BE1DB1F1'\r
+\r
+f = factories['ubuntu-packages'] = BuildFactory()\r
+# We need the git repository for the Debian packaging.\r
+f.addStep(s_git)\r
+unpack_dist_tarball(f)\r
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")\r
+\r
+# Add a new changelog entry to have the git version in the package version.\r
+cmd(f,\r
+ name = 'update changelog',\r
+ workdir = 'build/DIST',\r
+ command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],\r
+)\r
+\r
+cmd(f,\r
+ name = 'source pkg',\r
+ command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],\r
+ workdir = 'build/DIST',\r
+)\r
+\r
+for dist in distributions:\r
+ f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))\r
+\r
+# Create debian sid repository\r
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))\r
+f.addStep(transfer.StringDownload(\r
+ """Codename: precise\r
+Suite: unstable\r
+Architectures: i386 amd64 source\r
+Components: main\r
+DebIndices: Packages Release . .gz .bz2\r
+DscIndices: Sources Release . .gz .bz2\r
+SignWith: %(gpg_key)s\r
+""" % { "gpg_key": gpg_key },\r
+ slavedest = 'REPO-sid/conf/distributions',\r
+))\r
+\r
+# add source package to repository\r
+reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc')\r
+\r
+# Add keyring to the repository. We need to run git clone on our own because\r
+# the Git() step assumes there’s precisely one repository we want to deal with.\r
+# No big deal since the i3-autobuild-keyring repository is not big.\r
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')\r
+reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes')\r
+\r
+for dist in distributions:\r
+ # update the pbuilder\r
+ cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')\r
+\r
+ # build the package for each dist\r
+ f.addStep(ShellCommand(\r
+ logEnviron = False,\r
+ name = 'pkg ' + dist,\r
+ command = 'pbuilder-' + dist + ' build --binary-arch \\r
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',\r
+ warnOnFailure = True\r
+ ))\r
+\r
+ reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes')\r
+\r
+# upload the sid repo\r
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we\r
+# prefer providing old packages over providing no packages at all :).\r
+for directory in [ 'pool', 'dists' ]:\r
+ f.addStep(transfer.DirectoryUpload(\r
+ slavesrc = 'REPO-sid/' + directory,\r
+ masterdest = 'htdocs/ubuntu/precise/' + directory,\r
+ compress = 'bz2',\r
+ name = 'upload precise ' + directory,\r
+ haltOnFailure = True,\r
+ ))\r
+\r
+f.addStep(master.MasterShellCommand(\r
+ command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;",\r
+ name = 'cleanup old packages',\r
+))\r
+\r
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not\r
+# try to download Translation-*)\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ],\r
+ name = 'create i18n folder',\r
+))\r
+f.addStep(master.MasterShellCommand(\r
+ command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ],\r
+ name = 'touch i18n/Index',\r
+))\r
+\r
+\r
+c['builders'] = []\r
+\r
+# Add all builders to all buildslaves.\r
+for factoryname in factories.keys():\r
+ c['builders'].append(BuilderConfig(\r
+ name = factoryname,\r
+ slavenames=['docsteel-vm'],\r
+ factory=factories[factoryname],\r
+ ))\r
+\r
+\r
+####### STATUS TARGETS\r
+\r
+c['status'] = []\r
+\r
+authz_cfg=authz.Authz(\r
+ gracefulShutdown = False,\r
+ forceBuild = False,\r
+ forceAllBuilds = False,\r
+ pingBuilder = False,\r
+ stopBuild = False,\r
+ stopAllBuilds = False,\r
+ cancelPendingBuild = False,\r
+)\r
+\r
+c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))\r
+\r
+c['status'].append(buildbot.status.status_push.HttpStatusPush(\r
+ serverUrl = 'http://localhost:8080/push_buildbot',\r
+))\r
+\r
+####### PROJECT IDENTITY\r
+\r
+c['title'] = 'i3'\r
+c['titleURL'] = 'http://i3wm.org/'\r
+# Removed so that search engines don’t crawl it\r
+c['buildbotURL'] = 'http://localhost/'\r
+\r
+####### DB URL\r
+\r
+c['db'] = {\r
+ # This specifies what database buildbot uses to store its state. You can leave\r
+ # this at its default for all but the largest installations.\r
+ 'db_url' : "sqlite:///state.sqlite",\r
+}</tt></pre>\r
+</div></div>\r
+</div>\r
+</div>\r
+</div>\r
+<div id="footnotes"><hr /></div>\r
+<div id="footer" lang="de">\r
+© 2009-2011 Michael Stapelberg, <a href="/impress.html">Impressum</a>\r
+</div>\r
+</body>\r
+</html>\r
projects / i3 / i3.github.io / commitdiff