From: Michael Stapelberg Date: Sun, 5 Aug 2012 01:56:05 +0000 (+0200) Subject: add docs/buildbot describing the i3 buildbot setup X-Git-Url: https://git.sur5r.net/?a=commitdiff_plain;h=2e6ce98bb7086c9ff49d091600b9e86555ecac0e;p=i3%2Fi3.github.io add docs/buildbot describing the i3 buildbot setup --- diff --git a/_docs/Makefile b/_docs/Makefile index 9a02b2a..ba01bfe 100644 --- a/_docs/Makefile +++ b/_docs/Makefile @@ -1,5 +1,5 @@ -all: hacking-howto.html debugging.html debugging-release-version.html i3bar-protocol.html repositories.html userguide.html ipc.html multi-monitor.html wsbar.html i3status.html i3.html i3-config-wizard.html i3-nagbar.html i3-migrate-config-to-v4.html i3-msg.html testsuite.html +all: hacking-howto.html debugging.html debugging-release-version.html i3bar-protocol.html repositories.html buildbot.html userguide.html ipc.html multi-monitor.html wsbar.html i3status.html i3.html i3-config-wizard.html i3-nagbar.html i3-migrate-config-to-v4.html i3-msg.html testsuite.html hacking-howto.html: hacking-howto asciidoc -a linkcss -a stylesdir=/css -a scriptsdir=/js --backend=xhtml11 -f conf/i3html.conf -a toc -n $< @@ -19,6 +19,9 @@ i3bar-protocol.html: i3bar-protocol repositories.html: repositories asciidoc -a linkcss -a stylesdir=/css -a scriptsdir=/js --backend=xhtml11 -f conf/i3html.conf -n $< +buildbot.html: buildbot + asciidoc -a linkcss -a stylesdir=/css -a scriptsdir=/js --backend=xhtml11 -f conf/i3html.conf -a toc -n $< + userguide.html: userguide asciidoc -a linkcss -a stylesdir=/css -a scriptsdir=/js --backend=xhtml11 -f conf/i3html.conf -a toc -n $< diff --git a/_docs/buildbot b/_docs/buildbot new file mode 100644 index 0000000..77419e0 --- /dev/null +++ b/_docs/buildbot @@ -0,0 +1,1040 @@ +The i3 buildbot setup +===================== +Michael Stapelberg +August 2012 + +This document explains the http://www.buildbot.net/[buildbot] setup we use to +provide up-to-date documentation and debian packages at http://build.i3wm.org/. +We publish these information so that our setup is well-documented (thus +decreasing future maintenance effort) and because it might be interesting for +other projects. + +== Introduction + +What we are doing in i3 is called Continuous Integration (see +http://en.wikipedia.org/wiki/Continuous_integration): we publish the changes we +make on our local machines as often as possible. In order to maintain a +continuously high quality, each time any developer pushes changes to the +official git repository, a number of quality assurance tools start running +automatically: + +1. Latest documentation is generated and provided at + http://build.i3wm.org/docs/. This makes it easy to link to documentation for + features which are only in the current git version, not in the released + version. +2. The source code is compiled and it is automatically posted to the IRC + channel whether there were any compiler warnings. While developers should + notice compiler warnings, this mechanism creates a bit of public pressure + ("Oh, Michael introduced warnings with this commit!"). More importantly, + since this mechanism builds a dist tarball and then compiles that tarball, + any changes to the source which would result in an uncompilable dist tarball + are immediately obvious. Therefore, we could cut a release from the current + git version at any point in time. +3. The clang static analyzer runs and the resulting report is provided at + http://build.i3wm.org/clang-analyze/. While every developer needs to compile + his code before committing, he doesn’t necessarily use clang (so we catch + build failures when using clang) and he also probably doesn’t run a static + analyzer as part of his normal workflow. By just being available without any + friction, this mechanism encourages developers to look at the report and fix + problems. +4. Debian (and Ubuntu) packages are built. This not only ensures that we don’t + change anything in the source code which would lead to an FTBFS (Fails To + Build From Source) when building a Debian package, it also goes a long way + to encourage end users to test i3. To remove the need and resource + requirements for them to compile their own version of i3 regularly, we + provide packages that integrate conveniently with a normal Debian system + (e.g. that are automatically upgraded). + +== Why buildbot? + +Previously, I was unsatisfied with the current state of FOSS CI tools like +Jenkins, Tinderbox and others. They either seemed bloated, hard to use, +outdated or unappealing for some other reason. + +Then I discovered buildbot and was impressed by its flexibility. It let me +implement everything I wanted from a CI tool and (in my opinion) it is +light-weight, easy to deploy and well maintained. + +The only downside of buildbot is its configuration and documentation: You need +to spend quite a bit of time (I needed multiple days) until it works the way +you want it to and oftentimes, the documentation is far too sparse. This is one +of the reasons why I’m publishing the i3 setup. + +== Configuration + +See the next section for a complete, copy & pasteable configuration file. This +section covers the most important aspects without covering every line. + +This document assumes you are running buildbot 0.8.6p1. + +=== Change sources + +Since i3 uses a central git repository, we use the official buildbot +https://github.com/buildbot/buildbot/blob/master/master/contrib/git_buildbot.py[git +post-receive hook] that sends the change information to the buildbot master. + +=== Schedulers + +There are two things (called "builders" in buildbot-language) which happen +whenever a new change in the +next+ branch of i3 occurs: + +1. The "docs" builder builds and uploads the latest documentation. This happens + directly from the git repository with a custom asciidoc configuration which + indicates that these docs refer to the git version. Therefore, this builder + does not benefit from having a dist tarball available (contrary to the other + builders). + +2. The "dist" builder prepares a dist tarball and then triggers the remaining + builders. This ensures that building the dist tarball (an operation which + takes about one minute due to documentation generation) only happens once. + +Here is the relevant configuration part: + +*Schedulers*: +--------------------------------------------- +c['schedulers'] = [] + +c['schedulers'].append(SingleBranchScheduler( + name = 'dist', + branch = 'next', + treeStableTimer = 10, + builderNames = [ 'dist', 'docs' ], +)) + +c['schedulers'].append(Triggerable( + name = 'dist-tarball-done', + builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ], +)) +--------------------------------------------- + +=== Building the dist tarball + +This builder clones the i3 git repository and runs "make dist", which creates a +tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then +renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable +name in the next steps) and uploaded to the buildbot master (since we can have +multiple buildslaves, we cannot just let it rest on the buildslave that built +it). Afterwards, old dist tarballs are cleaned up and the remaining builders +are triggered: + +*Building a dist tarball*: +--------------------------------------------- +factories = {} + +f = factories['dist'] = BuildFactory() + +# Check out the git repository. +f.addStep(s_git) + +# Fill the 'gitversion' property with the output of git describe --tags. +f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion')) + +# Build the dist tarball. +cmd(f, name = 'make dist', command = [ 'make', 'dist' ]) + +# Rename the created tarball to a well-known name. +cmd(f, + name = 'rename tarball', + command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'), +) + +# Upload the dist tarball to the master (other factories download it later). +f.addStep(transfer.FileUpload( + slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'), + masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'), +)) + +# Cleanup old dist tarballs (everything older than tree days). +f.addStep(master.MasterShellCommand( + command = "find distballs -mtime +3 -exec rm '{}' \;", + name = 'cleanup old dist tarballs', +)) + +# Everything worked fine, now trigger compilation. +f.addStep(Trigger( + schedulerNames = [ 'dist-tarball-done' ], + copy_properties = [ 'gitversion' ], +)) +--------------------------------------------- + +Three things are noteworthy about this part of the configuration: + +1. For convenience, we call each factory +f+ (just like the global buildbot + config uses +c+ for the top-level configuration) and add it to a dictionary. + Factories in that dictionary are later automatically configured for each + buildslave. + +2. We have a shared step called +s_git+ so that we only have one location in + the configuration file where we specify the git repository URL and branch. + +3. We have a custom function called +cmd+ which is a shortcut for defining a + +ShellCommand+ with +haltOnFailure=True+ (since each step is critical) and + +logEnviron=False+ (for brevity). + +Here are their definitions: + +*cmd*: +--------------------------------------------- +def cmd(factory, **kwargs): + factory.addStep(ShellCommand( + haltOnFailure = True, + logEnviron = False, + **kwargs + )) +--------------------------------------------- + +*s_git*: +--------------------------------------------- +s_git = Git( + repourl = 'git://code.i3wm.org/i3', + branch = 'next', + + # Check out the latest revision, not the one which caused this build. + alwaysUseLatest = True, + + # We cannot use shallow because it breaks git describe --tags. + shallow = False, + + # Delete remnants of previous builds. + mode = 'full', + + # Store checkouts in source/ and copy them over to build/ to save + # bandwidth. + method = 'copy', +) +--------------------------------------------- + +=== Compiling the dist tarball + +For this builder to work, you obviously need to install all the +build-dependencies for your software on each buildslave. In the case of i3, +this can be done with +apt-get build-dep i3-wm+. + +The compilation is pretty straight-forward since it uses the builtin +Compile+ +step. We call +make+ with +-j4+ (we don’t have enough buildslaves to make +figuring out the amount of cores at build-time worthwhile) and +DEBUG=0+ to +simulate release build conditions. Also, we pass the preprocessor flag ++-D_FORTIFY_SOURCE=2+ and the compiler flags +-Wformat+ and +-Wformat-security+ +to enable additional warnings. + +*Compiling the dist tarball*: +--------------------------------------------- +f = factories['compile'] = BuildFactory() +unpack_dist_tarball(f) +f.addStep(Compile( + command = [ 'make', 'DEBUG=0', '-j4' ], + warningPattern = '.*warning: ', + warnOnWarnings = True, + workdir = 'build/DIST', + env = { + 'CPPFLAGS': '-D_FORTIFY_SOURCE=2', + 'CFLAGS': '-Wformat -Wformat-security' + }, +)) + +f.addStep(WarningsToIRC()) +--------------------------------------------- + +Again, we use custom functions (and a custom buildstep) to make our lives +easier. Here is the definition of unpack_dist_tarball which adds three steps to +the factory that download and unpack the dist tarball to the +DIST/+ directory: + +*unpack_dist_tarball*: +--------------------------------------------- +def unpack_dist_tarball(factory): + factory.addStep(transfer.FileDownload( + mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'), + slavedest = 'dist.tar.bz2', + )) + + factory.addStep(slave.MakeDirectory(dir = 'build/DIST')) + + cmd(factory, + name = 'unpack dist tarball', + command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ], + ) +--------------------------------------------- + +The +WarningsToIRC+ build step is a custom build step which sets a property +called "ircsuffix" that is used by our custom IRC bot. This is covered later in +more detail. This property gets set to a green or red message, depending on +whether there were any warnings: + +*WarningsToIRC*: +--------------------------------------------- +class WarningsToIRC(buildstep.BuildStep): + def start(self): + warnings = self.getProperty("warnings-count") + if warnings is not None and int(warnings) > 0: + warnings = int(warnings) # just to be sure + self.setProperty("ircsuffix", ("\0037 with %d warning%s!" % + (warnings, "s" if warnings != 1 else ""))) + else: + self.setProperty("ircsuffix", "\0033 without warnings") + self.finished(SUCCESS) +--------------------------------------------- + +=== Static code analysis + +For this builder to work, you additionally need the +clang+ compiler on each +buildslave: +apt-get install clang+. + +This builder uses only custom functions which you already know by now. It runs +scan-build, then moves scan-build’s output from a date-based directory directly +into the +CLANG/+ directory and uploads that to the buildmaster. + +On the buildmaster, a webserver is configured which has a symlink to ++/home/build/i3-master/htdocs/clang-analyze+ in its document root. + +*static code analysis*: +--------------------------------------------- +f = factories['clang-analyze'] = BuildFactory() +unpack_dist_tarball(f) +cmd(f, + name='analyze', + command = [ + 'scan-build', + '-o', '../CLANG', + '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'), + 'make', '-j8', + ], + workdir = 'build/DIST', +) + +# remove the subdirectory -- we always want to overwrite +cmd(f, command = 'mv CLANG/*/* CLANG/') + +f.addStep(transfer.DirectoryUpload( + slavesrc = 'CLANG', + masterdest = 'htdocs/clang-analyze', + compress = 'bz2', + name = 'upload output', +)) + +f.addStep(ClangToIRC()) +--------------------------------------------- + +The +ClangToIRC+ custom step is even simpler than +WarningsToIRC+. It simply +sets the ircsuffix property to a static message: + +*ClangToIRC*: +--------------------------------------------- +class ClangToIRC(buildstep.BuildStep): + def start(self): + self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/") + self.finished(SUCCESS) +--------------------------------------------- + +=== Generating documentation + +This builder is the one which is the least clean of all. It uses the Debian +packaging information to decide which docs to publish and which manpages to +generate. Additionally, it uses a for loop instead of calling a script. I +recommend including a script to do this in your repository instead. + +Apart from these concerns, the builder is straight-forward: It clones the git +repository, generates the documentation and then uploads the documentation to +the buildmaster: + +*Generating documentation*: +--------------------------------------------- +f = factories['docs'] = BuildFactory() +f.addStep(s_git) +# Fill the 'gitversion' property with the output of git describe --tags. +f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion')) +cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ]) +cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done") +f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS')) +cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS") +cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS") + +f.addStep(transfer.DirectoryUpload( + slavesrc = 'COPY-DOCS', + masterdest = 'htdocs/docs-git', + compress = 'bz2', + name = 'upload docs')) + +f.addStep(DocsToIRC()) +--------------------------------------------- + +Just as +ClangToIRC+, +DocsToIRC+ appends a static message: + +*DocsToIRC*: +--------------------------------------------- +class DocsToIRC(buildstep.BuildStep): + def start(self): + self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/") + self.finished(SUCCESS) +--------------------------------------------- + +=== Building Debian/Ubuntu packages + +This is the most complex builder of all. It uses +pbuilder-dist+, +debchange+, ++dpkg-buildpackage+ and +reprepro+ to generate a Debian repository with a +cleanly compiled package for amd64 and i386. In order for it to work, you need +to install the following packages: +apt-get install devscripts dpkg-dev +reprepro ubuntu-dev-tools+. Afterwards, you need to allow the user as which the +buildslave runs to execute pbuilder via sudo without needing a password, so run ++visudo+ and add a line like this one: + +*sudoers line*: +--------------------------------------------- +build ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder +--------------------------------------------- + +Then, as the user as which your buildslave runs, setup the pbuilder +environments (you only need to do this once): + +*pbuilder preparation*: +--------------------------------------------- +sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64 +sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386 +pbuilder-sid-amd64 create +pbuilder-sid-i386 create +--------------------------------------------- + +Also, you will need a GPG key to sign these packages. + +The debian builder starts by unpacking the dist tarball, copying the Debian +packaging from git, creating an empty Debian repository with the ++i3-autobuild-keyring+ contents in it. It then adds a new changelog entry to +reflect the git version and the fact that this package was built automatically, +builds a source package with +dpkg-buildpackage+ and adds it to the repository. +Afterwards, it updates each pbuilder and builds binary packages for each +architecture (amd64 and i386). After adding the resulting packages to the +repository, it uploads the repository to the buildmaster: + +*Debian builder*: +--------------------------------------------- +distributions = [ 'sid-amd64', 'sid-i386' ] +gpg_key = 'BE1DB1F1' + +f = factories['debian-packages'] = BuildFactory() +# We need the git repository for the Debian packaging. +f.addStep(s_git) +unpack_dist_tarball(f) +cmd(f, name = 'copy packaging', command = "cp -r debian DIST/") + +# Add a new changelog entry to have the git version in the package version. +cmd(f, + name = 'update changelog', + workdir = 'build/DIST', + command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ], +) + +cmd(f, + name = 'source pkg', + command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ], + workdir = 'build/DIST', +) + +for dist in distributions: + f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist)) + +# Create debian sid repository +f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf')) +f.addStep(transfer.StringDownload( + """Codename: sid +Suite: unstable +Architectures: i386 amd64 source +Components: main +DebIndices: Packages Release . .gz .bz2 +DscIndices: Sources Release . .gz .bz2 +SignWith: %(gpg_key)s +""" % { "gpg_key": gpg_key }, + slavedest = 'REPO-sid/conf/distributions', +)) + +# add source package to repository +reprepro_include(f, 'i3-wm*_source.changes', 'dsc') + +# Add keyring to the repository. We need to run git clone on our own because +# the Git() step assumes there’s precisely one repository we want to deal with. +# No big deal since the i3-autobuild-keyring repository is not big. +cmd(f, + name = 'clone keyring repo', + command = 'git clone git://code.i3wm.org/i3-autobuild-keyring', +) +reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes') + +for dist in distributions: + # update the pbuilder + cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update') + + # build the package for each dist + f.addStep(ShellCommand( + logEnviron = False, + name = 'pkg ' + dist, + command = 'pbuilder-' + dist + ' build --binary-arch \ +--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc', + warnOnFailure = True + )) + + reprepro_include(f, 'RESULT-' + dist + '/*.changes') + +# upload the sid repo +# Since the next step is cleaning up old files, we set haltOnFailure=True -- we +# prefer providing old packages over providing no packages at all :). +for directory in [ 'pool', 'dists' ]: + f.addStep(transfer.DirectoryUpload( + slavesrc = 'REPO-sid/' + directory, + masterdest = 'htdocs/debian/sid/' + directory, + compress = 'bz2', + name = 'upload sid ' + directory, + haltOnFailure = True, + )) + +f.addStep(master.MasterShellCommand( + command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;", + name = 'cleanup old packages', +)) + +# We ensure there is an empty i18n/Index to speed up apt (so that it does not +# try to download Translation-*) +f.addStep(master.MasterShellCommand( + command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ], + name = 'create i18n folder', +)) +f.addStep(master.MasterShellCommand( + command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ], + name = 'touch i18n/Index', +)) +--------------------------------------------- + +The +reprepro_include+ command is defined as follows: + +*reprepro_include*: +--------------------------------------------- +def reprepro_include(factory, path, debtype='deb', **kwargs): + cmd(factory, + name = 'reprepro include', + command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path, + **kwargs + ) +--------------------------------------------- + +Running such a builder for Ubuntu works exactly the same way, but you need to +replace "sid" with "precise" in all places (see the full configuration file for +an example). + +=== Status targets + +We don’t advertise the HTTP status target. Instead, status is posted to IRC via +a custom bot. This bot provides an HTTP end point and buildbot is configured to +push status changes to that endpoint: + +*http status target*: +--------------------------------------------- +c['status'].append(buildbot.status.status_push.HttpStatusPush( + serverUrl = 'http://localhost:8080/push_buildbot', +)) +--------------------------------------------- + +You can find the source code of that bot at +http://code.stapelberg.de/git/go-buildbot-announce/. As the name suggests, it +is written in Go. Also, it is quite specific to i3, so you might be better off +implementing such a bot (or plugin) on your own. It might make for a nice +example, though, especially back when its only feature was announcing the build +status: + +http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0 + +== Full configuration file + +This is the full configuration file, as tested and currently in use (except for +the passwords, though): + +*master.cfg*: +--------------------------------------------- +# -*- python -*- +# -*- coding: utf-8 +# vim:ts=4:sw=4:expandtab:syntax=python +# +# i3 buildbot configuration +# © 2012 Michael Stapelberg, Public Domain +# see http://i3wm.org/docs/buildbot.html for more information. + +from buildbot.buildslave import BuildSlave +from buildbot.changes import pb +from buildbot.schedulers.basic import SingleBranchScheduler +from buildbot.schedulers.triggerable import Triggerable +from buildbot.process.properties import WithProperties +from buildbot.process.factory import BuildFactory +from buildbot.steps.source.git import Git +from buildbot.steps.shell import ShellCommand +from buildbot.steps.shell import Compile +from buildbot.steps.trigger import Trigger +from buildbot.steps import shell, transfer, master, slave +from buildbot.config import BuilderConfig +from buildbot.process import buildstep +from buildbot.status import html +from buildbot.status import words +import buildbot.status.status_push +from buildbot.status.web import auth, authz +from buildbot.status.builder import SUCCESS, FAILURE + +c = BuildmasterConfig = {} + +c['slaves'] = [BuildSlave('docsteel-vm', 'secret')] +c['slavePortnum'] = 9989 +# Changes are pushed to buildbot using a git hook. +c['change_source'] = [pb.PBChangeSource( + user = 'i3-source', + passwd = 'secret', +)] + +################################################################################ +# schedulers +################################################################################ + +c['schedulers'] = [] + +# The first scheduler kicks off multiple builders: +# • 'dist' builds a dist tarball and starts the triggerable schedulers +# 'compile' +# • 'docs' builds the documentation with a special asciidoc configuration +# (therefore, it does not profit from a dist tarball and can be run in +# parallel). +c['schedulers'].append(SingleBranchScheduler( + name = 'dist', + branch = 'next', + treeStableTimer = 10, + builderNames = [ 'dist', 'docs' ], +)) + +c['schedulers'].append(Triggerable( + name = 'dist-tarball-done', + builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ], +)) + +################################################################################ +# Shortcuts for builders +################################################################################ + +# shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False +def cmd(factory, **kwargs): + factory.addStep(ShellCommand( + haltOnFailure=True, + logEnviron=False, + **kwargs + )) + +# Shortcut to add steps necessary to download and unpack the dist tarball. +def unpack_dist_tarball(factory): + factory.addStep(transfer.FileDownload( + mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'), + slavedest='dist.tar.bz2', + )) + factory.addStep(slave.MakeDirectory(dir='build/DIST')) + cmd(factory, + name = 'unpack dist tarball', + command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ], + ) + +# Includes the given path in REPO-sid using reprepro. +def reprepro_include(factory, path, debtype='deb', **kwargs): + cmd(factory, + name = 'reprepro include', + command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path, + **kwargs + ) + +def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs): + cmd(factory, + name = 'reprepro include', + command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path, + **kwargs + ) + +################################################################################ +# Custom steps +################################################################################ + +# Adds the ircsuffix property to reflect whether there were warnings. +class WarningsToIRC(buildstep.BuildStep): + def start(self): + warnings = self.getProperty("warnings-count") + if warnings is not None and int(warnings) > 0: + warnings = int(warnings) # just to be sure + self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else "")) + else: + self.setProperty("ircsuffix", "\0033 without warnings") + self.finished(SUCCESS) + +# Adds a link to the automatically generated documentation. +class DocsToIRC(buildstep.BuildStep): + def start(self): + self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/") + self.finished(SUCCESS) + +# Adds a link to the clang report. +class ClangToIRC(buildstep.BuildStep): + def start(self): + self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/") + self.finished(SUCCESS) + +################################################################################ +# Shared steps, used in different factories. +################################################################################ + +s_git = Git( + repourl='git://code.i3wm.org/i3', + branch='next', + + # Check out the latest revision, not the one which caused this build. + alwaysUseLatest=True, + + # We cannot use shallow because it breaks git describe --tags. + shallow=False, + + # Delete remnants of previous builds. + mode='full', + + # Store checkouts in source/ and copy them over to build/ to save + # bandwidth. + method='copy', + + # XXX: In newer versions of buildbot (> 0.8.6), we want to use + # getDescription={ 'tags': True } here and get rid of the extra git + # describe --tags step. +) + +################################################################################ +# factory: "dist" — builds the dist tarball once (used by all other factories) +################################################################################ + +factories = {} + +f = factories['dist'] = BuildFactory() +# Check out the git repository. +f.addStep(s_git) +# Fill the 'gitversion' property with the output of git describe --tags. +f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion')) +# Build the dist tarball. +cmd(f, name = 'make dist', command = [ 'make', 'dist' ]) +# Rename the created tarball to a well-known name. +cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2')) +# Upload the dist tarball to the master (other factories download it later). +f.addStep(transfer.FileUpload( + slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'), + masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'), +)) +# Cleanup old dist tarballs (everything older than tree days). +f.addStep(master.MasterShellCommand( + command = "find distballs -mtime +3 -exec rm '{}' \;", + name = 'cleanup old dist tarballs', +)) +# Everything worked fine, now trigger compilation. +f.addStep(Trigger( + schedulerNames = [ 'dist-tarball-done' ], + copy_properties = [ 'gitversion' ], +)) + +################################################################################ +# factory: "compile" — compiles the dist tarball and reports warnings +################################################################################ + +f = factories['compile'] = BuildFactory() +unpack_dist_tarball(f) +f.addStep(Compile( + command = [ 'make', 'DEBUG=0', '-j4' ], + warningPattern = '.*warning: ', + warnOnWarnings = True, + workdir = 'build/DIST', + env = { + 'CPPFLAGS': '-D_FORTIFY_SOURCE=2', + 'CFLAGS': '-Wformat -Wformat-security' + }, +)) + +f.addStep(WarningsToIRC()) + +################################################################################ +# factory: "clang-analyze" — runs a static code analysis +################################################################################ +# $ sudo apt-get install clang + +f = factories['clang-analyze'] = BuildFactory() +unpack_dist_tarball(f) +cmd(f, + name='analyze', + command = [ + 'scan-build', + '-o', '../CLANG', + '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'), + 'make', '-j8', + ], + workdir = 'build/DIST', +) + +# remove the subdirectory -- we always want to overwrite +cmd(f, command = 'mv CLANG/*/* CLANG/') + +f.addStep(transfer.DirectoryUpload( + slavesrc = 'CLANG', + masterdest = 'htdocs/clang-analyze', + compress = 'bz2', + name = 'upload output', +)) + +f.addStep(ClangToIRC()) + +################################################################################ +# factory: "docs" — builds documentation with a special asciidoc conf +################################################################################ + +f = factories['docs'] = BuildFactory() +f.addStep(s_git) +# Fill the 'gitversion' property with the output of git describe --tags. +f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion')) +cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ]) +cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done") +f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS')) +cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS") +cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS") + +f.addStep(transfer.DirectoryUpload( + slavesrc = 'COPY-DOCS', + masterdest = 'htdocs/docs-git', + compress = 'bz2', + name = 'upload docs')) + +f.addStep(DocsToIRC()) + +################################################################################ +# factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386 +################################################################################ + +distributions = [ 'sid-amd64', 'sid-i386' ] +gpg_key = 'BE1DB1F1' + +f = factories['debian-packages'] = BuildFactory() +# We need the git repository for the Debian packaging. +f.addStep(s_git) +unpack_dist_tarball(f) +cmd(f, name='copy packaging', command = "cp -r debian DIST/") + +# Add a new changelog entry to have the git version in the package version. +cmd(f, + name = 'update changelog', + workdir = 'build/DIST', + command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ], +) + +cmd(f, + name = 'source pkg', + command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ], + workdir = 'build/DIST', +) + +for dist in distributions: + f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist)) + +# Create debian sid repository +f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf')) +f.addStep(transfer.StringDownload( + """Codename: sid +Suite: unstable +Architectures: i386 amd64 source +Components: main +DebIndices: Packages Release . .gz .bz2 +DscIndices: Sources Release . .gz .bz2 +SignWith: %(gpg_key)s +""" % { "gpg_key": gpg_key }, + slavedest = 'REPO-sid/conf/distributions', +)) + +# add source package to repository +reprepro_include(f, 'i3-wm*_source.changes', 'dsc') + +# Add keyring to the repository. We need to run git clone on our own because +# the Git() step assumes there’s precisely one repository we want to deal with. +# No big deal since the i3-autobuild-keyring repository is not big. +cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring') +reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes') + +for dist in distributions: + # update the pbuilder + cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update') + + # build the package for each dist + f.addStep(ShellCommand( + logEnviron = False, + name = 'pkg ' + dist, + command = 'pbuilder-' + dist + ' build --binary-arch \ +--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc', + warnOnFailure = True + )) + + reprepro_include(f, 'RESULT-' + dist + '/*.changes') + +# upload the sid repo +# Since the next step is cleaning up old files, we set haltOnFailure=True -- we +# prefer providing old packages over providing no packages at all :). +for directory in [ 'pool', 'dists' ]: + f.addStep(transfer.DirectoryUpload( + slavesrc = 'REPO-sid/' + directory, + masterdest = 'htdocs/debian/sid/' + directory, + compress = 'bz2', + name = 'upload sid ' + directory, + haltOnFailure = True, + )) + +f.addStep(master.MasterShellCommand( + command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;", + name = 'cleanup old packages', +)) + +# We ensure there is an empty i18n/Index to speed up apt (so that it does not +# try to download Translation-*) +f.addStep(master.MasterShellCommand( + command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ], + name = 'create i18n folder', +)) +f.addStep(master.MasterShellCommand( + command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ], + name = 'touch i18n/Index', +)) + +################################################################################ +# factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386 +################################################################################ + +distributions = [ 'precise-amd64', 'precise-i386' ] +gpg_key = 'BE1DB1F1' + +f = factories['ubuntu-packages'] = BuildFactory() +# We need the git repository for the Debian packaging. +f.addStep(s_git) +unpack_dist_tarball(f) +cmd(f, name='copy packaging', command = "cp -r debian DIST/") + +# Add a new changelog entry to have the git version in the package version. +cmd(f, + name = 'update changelog', + workdir = 'build/DIST', + command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ], +) + +cmd(f, + name = 'source pkg', + command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ], + workdir = 'build/DIST', +) + +for dist in distributions: + f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist)) + +# Create debian sid repository +f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf')) +f.addStep(transfer.StringDownload( + """Codename: precise +Suite: unstable +Architectures: i386 amd64 source +Components: main +DebIndices: Packages Release . .gz .bz2 +DscIndices: Sources Release . .gz .bz2 +SignWith: %(gpg_key)s +""" % { "gpg_key": gpg_key }, + slavedest = 'REPO-sid/conf/distributions', +)) + +# add source package to repository +reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc') + +# Add keyring to the repository. We need to run git clone on our own because +# the Git() step assumes there’s precisely one repository we want to deal with. +# No big deal since the i3-autobuild-keyring repository is not big. +cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring') +reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes') + +for dist in distributions: + # update the pbuilder + cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update') + + # build the package for each dist + f.addStep(ShellCommand( + logEnviron = False, + name = 'pkg ' + dist, + command = 'pbuilder-' + dist + ' build --binary-arch \ +--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc', + warnOnFailure = True + )) + + reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes') + +# upload the sid repo +# Since the next step is cleaning up old files, we set haltOnFailure=True -- we +# prefer providing old packages over providing no packages at all :). +for directory in [ 'pool', 'dists' ]: + f.addStep(transfer.DirectoryUpload( + slavesrc = 'REPO-sid/' + directory, + masterdest = 'htdocs/ubuntu/precise/' + directory, + compress = 'bz2', + name = 'upload precise ' + directory, + haltOnFailure = True, + )) + +f.addStep(master.MasterShellCommand( + command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;", + name = 'cleanup old packages', +)) + +# We ensure there is an empty i18n/Index to speed up apt (so that it does not +# try to download Translation-*) +f.addStep(master.MasterShellCommand( + command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ], + name = 'create i18n folder', +)) +f.addStep(master.MasterShellCommand( + command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ], + name = 'touch i18n/Index', +)) + + +c['builders'] = [] + +# Add all builders to all buildslaves. +for factoryname in factories.keys(): + c['builders'].append(BuilderConfig( + name = factoryname, + slavenames=['docsteel-vm'], + factory=factories[factoryname], + )) + + +####### STATUS TARGETS + +c['status'] = [] + +authz_cfg=authz.Authz( + gracefulShutdown = False, + forceBuild = False, + forceAllBuilds = False, + pingBuilder = False, + stopBuild = False, + stopAllBuilds = False, + cancelPendingBuild = False, +) + +c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg)) + +c['status'].append(buildbot.status.status_push.HttpStatusPush( + serverUrl = 'http://localhost:8080/push_buildbot', +)) + +####### PROJECT IDENTITY + +c['title'] = 'i3' +c['titleURL'] = 'http://i3wm.org/' +# Removed so that search engines don’t crawl it +c['buildbotURL'] = 'http://localhost/' + +####### DB URL + +c['db'] = { + # This specifies what database buildbot uses to store its state. You can leave + # this at its default for all but the largest installations. + 'db_url' : "sqlite:///state.sqlite", +} +--------------------------------------------- diff --git a/docs/buildbot.html b/docs/buildbot.html new file mode 100644 index 0000000..786f1a2 --- /dev/null +++ b/docs/buildbot.html @@ -0,0 +1,1108 @@ + + + + + + +i3: The i3 buildbot setup + + + + + + + +
+

i3 - improved tiling WM

 + +
+
+ +
+
+

This document explains the buildbot setup we use to +provide up-to-date documentation and debian packages at http://build.i3wm.org/. +We publish these information so that our setup is well-documented (thus +decreasing future maintenance effort) and because it might be interesting for +other projects.

+
+
+
+

1. Introduction

+
+

What we are doing in i3 is called Continuous Integration (see +http://en.wikipedia.org/wiki/Continuous_integration): we publish the changes we +make on our local machines as often as possible. In order to maintain a +continuously high quality, each time any developer pushes changes to the +official git repository, a number of quality assurance tools start running +automatically:

+
    +
  1. +

    +Latest documentation is generated and provided at + http://build.i3wm.org/docs/. This makes it easy to link to documentation for + features which are only in the current git version, not in the released + version. +

    +
    2. +
  2. +

    +The source code is compiled and it is automatically posted to the IRC + channel whether there were any compiler warnings. While developers should + notice compiler warnings, this mechanism creates a bit of public pressure + ("Oh, Michael introduced warnings with this commit!"). More importantly, + since this mechanism builds a dist tarball and then compiles that tarball, + any changes to the source which would result in an uncompilable dist tarball + are immediately obvious. Therefore, we could cut a release from the current + git version at any point in time. +

    +
    3. +
  3. +

    +The clang static analyzer runs and the resulting report is provided at + http://build.i3wm.org/clang-analyze/. While every developer needs to compile + his code before committing, he doesn’t necessarily use clang (so we catch + build failures when using clang) and he also probably doesn’t run a static + analyzer as part of his normal workflow. By just being available without any + friction, this mechanism encourages developers to look at the report and fix + problems. +

    +
    4. +
  4. +

    +Debian (and Ubuntu) packages are built. This not only ensures that we don’t + change anything in the source code which would lead to an FTBFS (Fails To + Build From Source) when building a Debian package, it also goes a long way + to encourage end users to test i3. To remove the need and resource + requirements for them to compile their own version of i3 regularly, we + provide packages that integrate conveniently with a normal Debian system + (e.g. that are automatically upgraded). +

    +
    5. +
+
+
+
+

2. Why buildbot?

+
+

Previously, I was unsatisfied with the current state of FOSS CI tools like +Jenkins, Tinderbox and others. They either seemed bloated, hard to use, +outdated or unappealing for some other reason.

+

Then I discovered buildbot and was impressed by its flexibility. It let me +implement everything I wanted from a CI tool and (in my opinion) it is +light-weight, easy to deploy and well maintained.

+

The only downside of buildbot is its configuration and documentation: You need +to spend quite a bit of time (I needed multiple days) until it works the way +you want it to and oftentimes, the documentation is far too sparse. This is one +of the reasons why I’m publishing the i3 setup.

+
+
+
+

3. Configuration

+
+

See the next section for a complete, copy & pasteable configuration file. This +section covers the most important aspects without covering every line.

+

This document assumes you are running buildbot 0.8.6p1.

+
+

3.1. Change sources

+

Since i3 uses a central git repository, we use the official buildbot +git +post-receive hook that sends the change information to the buildbot master.

+
+
+

3.2. Schedulers

+

There are two things (called "builders" in buildbot-language) which happen +whenever a new change in the next branch of i3 occurs:

+
    +
  1. +

    +The "docs" builder builds and uploads the latest documentation. This happens + directly from the git repository with a custom asciidoc configuration which + indicates that these docs refer to the git version. Therefore, this builder + does not benefit from having a dist tarball available (contrary to the other + builders). +

    +
    2. +
  2. +

    +The "dist" builder prepares a dist tarball and then triggers the remaining + builders. This ensures that building the dist tarball (an operation which + takes about one minute due to documentation generation) only happens once. +

    +
    3. +
+

Here is the relevant configuration part:

+

Schedulers:

+
+
+
c['schedulers'] = []
+
+c['schedulers'].append(SingleBranchScheduler(
+    name = 'dist',
+    branch = 'next',
+    treeStableTimer = 10,
+    builderNames = [ 'dist', 'docs' ],
+))
+
+c['schedulers'].append(Triggerable(
+    name = 'dist-tarball-done',
+    builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
+))
+
+
+
+

3.3. Building the dist tarball

+

This builder clones the i3 git repository and runs "make dist", which creates a +tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then +renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable +name in the next steps) and uploaded to the buildbot master (since we can have +multiple buildslaves, we cannot just let it rest on the buildslave that built +it). Afterwards, old dist tarballs are cleaned up and the remaining builders +are triggered:

+

Building a dist tarball:

+
+
+
factories = {}
+
+f = factories['dist'] = BuildFactory()
+
+# Check out the git repository.
+f.addStep(s_git)
+
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+
+# Build the dist tarball.
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
+
+# Rename the created tarball to a well-known name.
+cmd(f,
+    name = 'rename tarball',
+    command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'),
+)
+
+# Upload the dist tarball to the master (other factories download it later).
+f.addStep(transfer.FileUpload(
+    slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
+    masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+))
+
+# Cleanup old dist tarballs (everything older than tree days).
+f.addStep(master.MasterShellCommand(
+    command = "find distballs -mtime +3 -exec rm '{}' \;",
+    name = 'cleanup old dist tarballs',
+))
+
+# Everything worked fine, now trigger compilation.
+f.addStep(Trigger(
+    schedulerNames = [ 'dist-tarball-done' ],
+    copy_properties = [ 'gitversion' ],
+))
+
+

Three things are noteworthy about this part of the configuration:

+
    +
  1. +

    +For convenience, we call each factory f (just like the global buildbot + config uses c for the top-level configuration) and add it to a dictionary. + Factories in that dictionary are later automatically configured for each + buildslave. +

    +
    2. +
  2. +

    +We have a shared step called s_git so that we only have one location in + the configuration file where we specify the git repository URL and branch. +

    +
    3. +
  3. +

    +We have a custom function called cmd which is a shortcut for defining a + ShellCommand with haltOnFailure=True (since each step is critical) and + logEnviron=False (for brevity). +

    +
    4. +
+

Here are their definitions:

+

cmd:

+
+
+
def cmd(factory, **kwargs):
+    factory.addStep(ShellCommand(
+        haltOnFailure = True,
+        logEnviron = False,
+        **kwargs
+    ))
+
+

s_git:

+
+
+
s_git = Git(
+    repourl = 'git://code.i3wm.org/i3',
+    branch = 'next',
+
+    # Check out the latest revision, not the one which caused this build.
+    alwaysUseLatest = True,
+
+    # We cannot use shallow because it breaks git describe --tags.
+    shallow = False,
+
+    # Delete remnants of previous builds.
+    mode = 'full',
+
+    # Store checkouts in source/ and copy them over to build/ to save
+    # bandwidth.
+    method = 'copy',
+)
+
+
+
+

3.4. Compiling the dist tarball

+

For this builder to work, you obviously need to install all the +build-dependencies for your software on each buildslave. In the case of i3, +this can be done with apt-get build-dep i3-wm.

+

The compilation is pretty straight-forward since it uses the builtin Compile +step. We call make with -j4 (we don’t have enough buildslaves to make +figuring out the amount of cores at build-time worthwhile) and DEBUG=0 to +simulate release build conditions. Also, we pass the preprocessor flag +-D_FORTIFY_SOURCE=2 and the compiler flags -Wformat and -Wformat-security +to enable additional warnings.

+

Compiling the dist tarball:

+
+
+
f = factories['compile'] = BuildFactory()
+unpack_dist_tarball(f)
+f.addStep(Compile(
+    command = [ 'make', 'DEBUG=0', '-j4' ],
+    warningPattern = '.*warning: ',
+    warnOnWarnings = True,
+    workdir = 'build/DIST',
+    env = {
+      'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
+      'CFLAGS': '-Wformat -Wformat-security'
+    },
+))
+
+f.addStep(WarningsToIRC())
+
+

Again, we use custom functions (and a custom buildstep) to make our lives +easier. Here is the definition of unpack_dist_tarball which adds three steps to +the factory that download and unpack the dist tarball to the DIST/ directory:

+

unpack_dist_tarball:

+
+
+
def unpack_dist_tarball(factory):
+    factory.addStep(transfer.FileDownload(
+        mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+        slavedest = 'dist.tar.bz2',
+    ))
+
+    factory.addStep(slave.MakeDirectory(dir = 'build/DIST'))
+
+    cmd(factory,
+        name = 'unpack dist tarball',
+        command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
+    )
+
+

The WarningsToIRC build step is a custom build step which sets a property +called "ircsuffix" that is used by our custom IRC bot. This is covered later in +more detail. This property gets set to a green or red message, depending on +whether there were any warnings:

+

WarningsToIRC:

+
+
+
class WarningsToIRC(buildstep.BuildStep):
+    def start(self):
+        warnings = self.getProperty("warnings-count")
+        if warnings is not None and int(warnings) > 0:
+            warnings = int(warnings)  # just to be sure
+            self.setProperty("ircsuffix", ("\0037 with %d warning%s!" %
+                (warnings, "s" if warnings != 1 else "")))
+        else:
+            self.setProperty("ircsuffix", "\0033 without warnings")
+        self.finished(SUCCESS)
+
+
+
+

3.5. Static code analysis

+

For this builder to work, you additionally need the clang compiler on each +buildslave: apt-get install clang.

+

This builder uses only custom functions which you already know by now. It runs +scan-build, then moves scan-build’s output from a date-based directory directly +into the CLANG/ directory and uploads that to the buildmaster.

+

On the buildmaster, a webserver is configured which has a symlink to +/home/build/i3-master/htdocs/clang-analyze in its document root.

+

static code analysis:

+
+
+
f = factories['clang-analyze'] = BuildFactory()
+unpack_dist_tarball(f)
+cmd(f,
+    name='analyze',
+    command = [
+        'scan-build',
+        '-o', '../CLANG',
+        '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
+        'make', '-j8',
+    ],
+    workdir = 'build/DIST',
+)
+
+# remove the subdirectory -- we always want to overwrite
+cmd(f, command = 'mv CLANG/*/* CLANG/')
+
+f.addStep(transfer.DirectoryUpload(
+    slavesrc = 'CLANG',
+    masterdest = 'htdocs/clang-analyze',
+    compress = 'bz2',
+    name = 'upload output',
+))
+
+f.addStep(ClangToIRC())
+
+

The ClangToIRC custom step is even simpler than WarningsToIRC. It simply +sets the ircsuffix property to a static message:

+

ClangToIRC:

+
+
+
class ClangToIRC(buildstep.BuildStep):
+    def start(self):
+        self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
+        self.finished(SUCCESS)
+
+
+
+

3.6. Generating documentation

+

This builder is the one which is the least clean of all. It uses the Debian +packaging information to decide which docs to publish and which manpages to +generate. Additionally, it uses a for loop instead of calling a script. I +recommend including a script to do this in your repository instead.

+

Apart from these concerns, the builder is straight-forward: It clones the git +repository, generates the documentation and then uploads the documentation to +the buildmaster:

+

Generating documentation:

+
+
+
f = factories['docs'] = BuildFactory()
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
+
+f.addStep(transfer.DirectoryUpload(
+    slavesrc = 'COPY-DOCS',
+    masterdest = 'htdocs/docs-git',
+    compress = 'bz2',
+    name = 'upload docs'))
+
+f.addStep(DocsToIRC())
+
+

Just as ClangToIRC, DocsToIRC appends a static message:

+

DocsToIRC:

+
+
+
class DocsToIRC(buildstep.BuildStep):
+    def start(self):
+        self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
+        self.finished(SUCCESS)
+
+
+
+

3.7. Building Debian/Ubuntu packages

+

This is the most complex builder of all. It uses pbuilder-dist, debchange, +dpkg-buildpackage and reprepro to generate a Debian repository with a +cleanly compiled package for amd64 and i386. In order for it to work, you need +to install the following packages: apt-get install devscripts dpkg-dev +reprepro ubuntu-dev-tools. Afterwards, you need to allow the user as which the +buildslave runs to execute pbuilder via sudo without needing a password, so run +visudo and add a line like this one:

+

sudoers line:

+
+
+
build    ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder
+
+

Then, as the user as which your buildslave runs, setup the pbuilder +environments (you only need to do this once):

+

pbuilder preparation:

+
+
+
sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64
+sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386
+pbuilder-sid-amd64 create
+pbuilder-sid-i386 create
+
+

Also, you will need a GPG key to sign these packages.

+

The debian builder starts by unpacking the dist tarball, copying the Debian +packaging from git, creating an empty Debian repository with the +i3-autobuild-keyring contents in it. It then adds a new changelog entry to +reflect the git version and the fact that this package was built automatically, +builds a source package with dpkg-buildpackage and adds it to the repository. +Afterwards, it updates each pbuilder and builds binary packages for each +architecture (amd64 and i386). After adding the resulting packages to the +repository, it uploads the repository to the buildmaster:

+

Debian builder:

+
+
+
distributions = [ 'sid-amd64', 'sid-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['debian-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name = 'copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+    name = 'update changelog',
+    workdir = 'build/DIST',
+    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+    name = 'source pkg',
+    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+    workdir = 'build/DIST',
+)
+
+for dist in distributions:
+    f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+    """Codename: sid
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+    slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f,
+    name = 'clone keyring repo',
+    command = 'git clone git://code.i3wm.org/i3-autobuild-keyring',
+)
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+    # update the pbuilder
+    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+    # build the package for each dist
+    f.addStep(ShellCommand(
+        logEnviron = False,
+        name = 'pkg ' + dist,
+        command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+        warnOnFailure = True
+    ))
+
+    reprepro_include(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+    f.addStep(transfer.DirectoryUpload(
+        slavesrc = 'REPO-sid/' + directory,
+        masterdest = 'htdocs/debian/sid/' + directory,
+        compress = 'bz2',
+        name = 'upload sid ' + directory,
+        haltOnFailure = True,
+    ))
+
+f.addStep(master.MasterShellCommand(
+    command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
+    name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+    command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
+    name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+    command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
+    name = 'touch i18n/Index',
+))
+
+

The reprepro_include command is defined as follows:

+

reprepro_include:

+
+
+
def reprepro_include(factory, path, debtype='deb', **kwargs):
+    cmd(factory,
+        name = 'reprepro include',
+        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
+        **kwargs
+    )
+
+

Running such a builder for Ubuntu works exactly the same way, but you need to +replace "sid" with "precise" in all places (see the full configuration file for +an example).

+
+
+

3.8. Status targets

+

We don’t advertise the HTTP status target. Instead, status is posted to IRC via +a custom bot. This bot provides an HTTP end point and buildbot is configured to +push status changes to that endpoint:

+

http status target:

+
+
+
c['status'].append(buildbot.status.status_push.HttpStatusPush(
+    serverUrl = 'http://localhost:8080/push_buildbot',
+))
+
+

You can find the source code of that bot at +http://code.stapelberg.de/git/go-buildbot-announce/. As the name suggests, it +is written in Go. Also, it is quite specific to i3, so you might be better off +implementing such a bot (or plugin) on your own. It might make for a nice +example, though, especially back when its only feature was announcing the build +status:

+

http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0

+
+
+
+
+

4. Full configuration file

+
+

This is the full configuration file, as tested and currently in use (except for +the passwords, though):

+

master.cfg:

+
+
+
# -*- python -*-
+# -*- coding: utf-8
+# vim:ts=4:sw=4:expandtab:syntax=python
+#
+# i3 buildbot configuration
+# © 2012 Michael Stapelberg, Public Domain
+# see http://i3wm.org/docs/buildbot.html for more information.
+
+from buildbot.buildslave import BuildSlave
+from buildbot.changes import pb
+from buildbot.schedulers.basic import SingleBranchScheduler
+from buildbot.schedulers.triggerable import Triggerable
+from buildbot.process.properties import WithProperties
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.source.git import Git
+from buildbot.steps.shell import ShellCommand
+from buildbot.steps.shell import Compile
+from buildbot.steps.trigger import Trigger
+from buildbot.steps import shell, transfer, master, slave
+from buildbot.config import BuilderConfig
+from buildbot.process import buildstep
+from buildbot.status import html
+from buildbot.status import words
+import buildbot.status.status_push
+from buildbot.status.web import auth, authz
+from buildbot.status.builder import SUCCESS, FAILURE
+
+c = BuildmasterConfig = {}
+
+c['slaves'] = [BuildSlave('docsteel-vm', 'secret')]
+c['slavePortnum'] = 9989
+# Changes are pushed to buildbot using a git hook.
+c['change_source'] = [pb.PBChangeSource(
+    user = 'i3-source',
+    passwd = 'secret',
+)]
+
+################################################################################
+# schedulers
+################################################################################
+
+c['schedulers'] = []
+
+# The first scheduler kicks off multiple builders:
+# • 'dist' builds a dist tarball and starts the triggerable schedulers
+#   'compile'
+# • 'docs' builds the documentation with a special asciidoc configuration
+#   (therefore, it does not profit from a dist tarball and can be run in
+#    parallel).
+c['schedulers'].append(SingleBranchScheduler(
+    name = 'dist',
+    branch = 'next',
+    treeStableTimer = 10,
+    builderNames = [ 'dist', 'docs' ],
+))
+
+c['schedulers'].append(Triggerable(
+    name = 'dist-tarball-done',
+    builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
+))
+
+################################################################################
+# Shortcuts for builders
+################################################################################
+
+# shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False
+def cmd(factory, **kwargs):
+    factory.addStep(ShellCommand(
+        haltOnFailure=True,
+        logEnviron=False,
+        **kwargs
+    ))
+
+# Shortcut to add steps necessary to download and unpack the dist tarball.
+def unpack_dist_tarball(factory):
+    factory.addStep(transfer.FileDownload(
+        mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+        slavedest='dist.tar.bz2',
+    ))
+    factory.addStep(slave.MakeDirectory(dir='build/DIST'))
+    cmd(factory,
+        name = 'unpack dist tarball',
+        command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
+    )
+
+# Includes the given path in REPO-sid using reprepro.
+def reprepro_include(factory, path, debtype='deb', **kwargs):
+    cmd(factory,
+        name = 'reprepro include',
+        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
+        **kwargs
+    )
+
+def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs):
+    cmd(factory,
+        name = 'reprepro include',
+        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path,
+        **kwargs
+    )
+
+################################################################################
+# Custom steps
+################################################################################
+
+# Adds the ircsuffix property to reflect whether there were warnings.
+class WarningsToIRC(buildstep.BuildStep):
+  def start(self):
+    warnings = self.getProperty("warnings-count")
+    if warnings is not None and int(warnings) > 0:
+      warnings = int(warnings)  # just to be sure
+      self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else ""))
+    else:
+      self.setProperty("ircsuffix", "\0033 without warnings")
+    self.finished(SUCCESS)
+
+# Adds a link to the automatically generated documentation.
+class DocsToIRC(buildstep.BuildStep):
+  def start(self):
+    self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
+    self.finished(SUCCESS)
+
+# Adds a link to the clang report.
+class ClangToIRC(buildstep.BuildStep):
+  def start(self):
+    self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
+    self.finished(SUCCESS)
+
+################################################################################
+# Shared steps, used in different factories.
+################################################################################
+
+s_git = Git(
+    repourl='git://code.i3wm.org/i3',
+    branch='next',
+
+    # Check out the latest revision, not the one which caused this build.
+    alwaysUseLatest=True,
+
+    # We cannot use shallow because it breaks git describe --tags.
+    shallow=False,
+
+    # Delete remnants of previous builds.
+    mode='full',
+
+    # Store checkouts in source/ and copy them over to build/ to save
+    # bandwidth.
+    method='copy',
+
+    # XXX: In newer versions of buildbot (> 0.8.6), we want to use
+    # getDescription={ 'tags': True } here and get rid of the extra git
+    # describe --tags step.
+)
+
+################################################################################
+# factory: "dist" — builds the dist tarball once (used by all other factories)
+################################################################################
+
+factories = {}
+
+f = factories['dist'] = BuildFactory()
+# Check out the git repository.
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+# Build the dist tarball.
+cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
+# Rename the created tarball to a well-known name.
+cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'))
+# Upload the dist tarball to the master (other factories download it later).
+f.addStep(transfer.FileUpload(
+    slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
+    masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
+))
+# Cleanup old dist tarballs (everything older than tree days).
+f.addStep(master.MasterShellCommand(
+    command = "find distballs -mtime +3 -exec rm '{}' \;",
+    name = 'cleanup old dist tarballs',
+))
+# Everything worked fine, now trigger compilation.
+f.addStep(Trigger(
+    schedulerNames = [ 'dist-tarball-done' ],
+    copy_properties = [ 'gitversion' ],
+))
+
+################################################################################
+# factory: "compile" — compiles the dist tarball and reports warnings
+################################################################################
+
+f = factories['compile'] = BuildFactory()
+unpack_dist_tarball(f)
+f.addStep(Compile(
+    command = [ 'make', 'DEBUG=0', '-j4' ],
+    warningPattern = '.*warning: ',
+    warnOnWarnings = True,
+    workdir = 'build/DIST',
+    env = {
+      'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
+      'CFLAGS': '-Wformat -Wformat-security'
+    },
+))
+
+f.addStep(WarningsToIRC())
+
+################################################################################
+# factory: "clang-analyze" — runs a static code analysis
+################################################################################
+# $ sudo apt-get install clang
+
+f = factories['clang-analyze'] = BuildFactory()
+unpack_dist_tarball(f)
+cmd(f,
+    name='analyze',
+    command = [
+        'scan-build',
+        '-o', '../CLANG',
+        '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
+        'make', '-j8',
+    ],
+    workdir = 'build/DIST',
+)
+
+# remove the subdirectory -- we always want to overwrite
+cmd(f, command = 'mv CLANG/*/* CLANG/')
+
+f.addStep(transfer.DirectoryUpload(
+    slavesrc = 'CLANG',
+    masterdest = 'htdocs/clang-analyze',
+    compress = 'bz2',
+    name = 'upload output',
+))
+
+f.addStep(ClangToIRC())
+
+################################################################################
+# factory: "docs" — builds documentation with a special asciidoc conf
+################################################################################
+
+f = factories['docs'] = BuildFactory()
+f.addStep(s_git)
+# Fill the 'gitversion' property with the output of git describe --tags.
+f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
+cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
+cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
+f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
+cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
+cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")
+
+f.addStep(transfer.DirectoryUpload(
+    slavesrc = 'COPY-DOCS',
+    masterdest = 'htdocs/docs-git',
+    compress = 'bz2',
+    name = 'upload docs'))
+
+f.addStep(DocsToIRC())
+
+################################################################################
+# factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386
+################################################################################
+
+distributions = [ 'sid-amd64', 'sid-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['debian-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+    name = 'update changelog',
+    workdir = 'build/DIST',
+    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+    name = 'source pkg',
+    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+    workdir = 'build/DIST',
+)
+
+for dist in distributions:
+    f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+    """Codename: sid
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+    slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
+reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+    # update the pbuilder
+    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+    # build the package for each dist
+    f.addStep(ShellCommand(
+        logEnviron = False,
+        name = 'pkg ' + dist,
+        command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+        warnOnFailure = True
+    ))
+
+    reprepro_include(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+    f.addStep(transfer.DirectoryUpload(
+        slavesrc = 'REPO-sid/' + directory,
+        masterdest = 'htdocs/debian/sid/' + directory,
+        compress = 'bz2',
+        name = 'upload sid ' + directory,
+        haltOnFailure = True,
+    ))
+
+f.addStep(master.MasterShellCommand(
+    command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
+    name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+    command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
+    name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+    command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
+    name = 'touch i18n/Index',
+))
+
+################################################################################
+# factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386
+################################################################################
+
+distributions = [ 'precise-amd64', 'precise-i386' ]
+gpg_key = 'BE1DB1F1'
+
+f = factories['ubuntu-packages'] = BuildFactory()
+# We need the git repository for the Debian packaging.
+f.addStep(s_git)
+unpack_dist_tarball(f)
+cmd(f, name='copy packaging', command = "cp -r debian DIST/")
+
+# Add a new changelog entry to have the git version in the package version.
+cmd(f,
+    name = 'update changelog',
+    workdir = 'build/DIST',
+    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
+)
+
+cmd(f,
+    name = 'source pkg',
+    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
+    workdir = 'build/DIST',
+)
+
+for dist in distributions:
+    f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))
+
+# Create debian sid repository
+f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
+f.addStep(transfer.StringDownload(
+    """Codename: precise
+Suite: unstable
+Architectures: i386 amd64 source
+Components: main
+DebIndices: Packages Release . .gz .bz2
+DscIndices: Sources Release . .gz .bz2
+SignWith: %(gpg_key)s
+""" % { "gpg_key": gpg_key },
+    slavedest = 'REPO-sid/conf/distributions',
+))
+
+# add source package to repository
+reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc')
+
+# Add keyring to the repository. We need to run git clone on our own because
+# the Git() step assumes there’s precisely one repository we want to deal with.
+# No big deal since the i3-autobuild-keyring repository is not big.
+cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
+reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes')
+
+for dist in distributions:
+    # update the pbuilder
+    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')
+
+    # build the package for each dist
+    f.addStep(ShellCommand(
+        logEnviron = False,
+        name = 'pkg ' + dist,
+        command = 'pbuilder-' + dist + ' build --binary-arch \
+--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
+        warnOnFailure = True
+    ))
+
+    reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes')
+
+# upload the sid repo
+# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
+# prefer providing old packages over providing no packages at all :).
+for directory in [ 'pool', 'dists' ]:
+    f.addStep(transfer.DirectoryUpload(
+        slavesrc = 'REPO-sid/' + directory,
+        masterdest = 'htdocs/ubuntu/precise/' + directory,
+        compress = 'bz2',
+        name = 'upload precise ' + directory,
+        haltOnFailure = True,
+    ))
+
+f.addStep(master.MasterShellCommand(
+    command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;",
+    name = 'cleanup old packages',
+))
+
+# We ensure there is an empty i18n/Index to speed up apt (so that it does not
+# try to download Translation-*)
+f.addStep(master.MasterShellCommand(
+    command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ],
+    name = 'create i18n folder',
+))
+f.addStep(master.MasterShellCommand(
+    command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ],
+    name = 'touch i18n/Index',
+))
+
+
+c['builders'] = []
+
+# Add all builders to all buildslaves.
+for factoryname in factories.keys():
+    c['builders'].append(BuilderConfig(
+        name = factoryname,
+        slavenames=['docsteel-vm'],
+        factory=factories[factoryname],
+    ))
+
+
+####### STATUS TARGETS
+
+c['status'] = []
+
+authz_cfg=authz.Authz(
+    gracefulShutdown = False,
+    forceBuild = False,
+    forceAllBuilds = False,
+    pingBuilder = False,
+    stopBuild = False,
+    stopAllBuilds = False,
+    cancelPendingBuild = False,
+)
+
+c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
+
+c['status'].append(buildbot.status.status_push.HttpStatusPush(
+    serverUrl = 'http://localhost:8080/push_buildbot',
+))
+
+####### PROJECT IDENTITY
+
+c['title'] = 'i3'
+c['titleURL'] = 'http://i3wm.org/'
+# Removed so that search engines don’t crawl it
+c['buildbotURL'] = 'http://localhost/'
+
+####### DB URL
+
+c['db'] = {
+    # This specifies what database buildbot uses to store its state.  You can leave
+    # this at its default for all but the largest installations.
+    'db_url' : "sqlite:///state.sqlite",
+}
+
+
+
+
+
+ + + diff --git a/docs/index.html.mako b/docs/index.html.mako index e70c804..0fc11a1 100644 --- a/docs/index.html.mako +++ b/docs/index.html.mako @@ -79,6 +79,12 @@ case. Documents the JSON based protocol which i3bar uses.

+

+i3 buildbot setup
+Describes the buildbot setup we use for +automatic docs, compilation and debian packages. +

+