i3 testsuite
============
-Michael Stapelberg <michael+i3@stapelberg.de>
-October 2011
+Michael Stapelberg <michael@i3wm.org>
+September 2012
This document explains how the i3 testsuite works, how to use it and extend it.
It is targeted at developers who not necessarily have been doing testing before
test manually if everything works. Having a testcase not only helps you with
that, but it will also be useful for every future change.
+== Relevant documentation
+
+Apart from this document, you should also have a look at:
+
+1. The "Modern Perl" book, which can be found at
+ http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf
+2. The latest Perl documentation of the "i3test" (general testcase setup) and
+ "i3test::Test" (additional test instructions) modules:
+ https://build.i3wm.org/docs/lib-i3test.html respectively
+ https://build.i3wm.org/docs/lib-i3test-test.html
+3. The latest documentation on i3’s IPC interface:
+ https://build.i3wm.org/docs/ipc.html
+
== Implementation
For several reasons, the i3 testsuite has been implemented in Perl:
2. Perl is widely available and has a well-working package infrastructure.
3. The author is familiar with Perl :).
+4. It is a good idea to use a different language for the tests than the
+ implementation itself.
Please do not start programming language flamewars at this point.
+=== Installing the dependencies
+
+As usual with Perl programs, the testsuite ships with a +Makefile.PL+.
+This file specifies which Perl modules the testsuite depends on and can be used
+to install all of them.
+
+Perl modules are distributed via CPAN, and there is the official, standard CPAN
+client, simply called +cpan+. It comes with every Perl installation and can be
+used to install the testsuite. Many users prefer to use the more modern
++cpanminus+ instead, though (because it asks no questions and just works):
+
+The tests additionally require +Xephyr(1)+ to run a nested X server. Install
++xserver-xephyr+ on Debian or +xorg-server-xephyr+ on Arch Linux.
+
+.Installing testsuite dependencies using cpanminus (preferred)
+--------------------------------------------------------------------------------
+$ cd ~/i3/testcases
+$ sudo apt-get install cpanminus
+$ sudo cpanm .
+$ cd ~/i3/AnyEvent-I3
+$ sudo cpanm Module::Install
+$ sudo cpanm .
+--------------------------------------------------------------------------------
+
+If you don’t want to use cpanminus for some reason, the same works with cpan:
+
+.Installing testsuite dependencies using cpan
+--------------------------------------------------------------------------------
+$ cd ~/i3/testcases
+$ sudo cpan .
+$ cd ~/i3/AnyEvent-I3
+$ sudo cpan Module::Install
+$ sudo cpan .
+--------------------------------------------------------------------------------
+
+In case you don’t have root permissions, you can also install into your home
+directory, see https://michael.stapelberg.de/cpan/
+
=== Mechanisms
==== Script: complete-run
testcases by default, but you can be more specific and let it only run one or
more testcases. Also, it takes care of starting up a separate instance of i3
with an appropriate configuration file and creates a folder for each run
-containing the appropriate i3 logfile for each testcase. The latest folder can
-always be found under the symlink +latest/+. It is recommended that you run the
-tests on one or more separate X server instances (you can only start one window
-manager per X session), for example using the provided Xdummy script.
-+complete-run.pl+ takes one or more X11 display specifications and parallelizes
-the testcases appropriately:
-
-.Example invocation of complete-run.pl+
+containing the appropriate i3 logfile for each testcase. The latest folder can
+always be found under the symlink +latest/+. Unless told differently, it will
+run the tests on a separate X server instance (using Xephyr).
+
+Xephyr will open a window where you can inspect the running test. By default,
+tests are run under Xvfb.
+
+.Example invocation of +complete-run.pl+
---------------------------------------
-$ cd ~/i3/testcases
+$ cd ~/i3
+
+$ autoreconf -fi
+
+$ mkdir -p build && cd build
+
+$ ../configure
+
+$ make -j8
+# output omitted because it is very long
-# start two dummy X11 instances in the background
-$ ./Xdummy :1 &
-$ ./Xdummy :2 &
+$ cd testcases
-$ ./complete-run.pl -d :1,:2
+$ ./complete-run.pl
# output omitted because it is very long
All tests successful.
Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU)
Result: PASS
-$ ./complete-run.pl -d :1 t/04-floating.t
+$ ./complete-run.pl t/04-floating.t
[:3] i3 startup: took 0.07s, status = 1
[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
[:3] t/04-floating.t finished
$ less latest/i3-log-for-04-floating.t
----------------------------------------
+If your attempt to run the tests with a bare call to ./complete-run.pl fails, try this:
+
+---------------------------------------------------
+$ ./complete-run.pl --parallel=1 --keep-xserver-output
+---------------------------------------------------
+
+This will show the output of Xephyr, which is the X server implementation we
+use for testing.
+
+===== make command: +make check+
+Make check runs the i3 testsuite.
+You can still use ./testcases/complete-run.pl to get the interactive progress output.
+
+.Example invocation of +make check+
+---------------------------------------
+$ cd ~/i3
+
+$ autoreconf -fi
+
+$ mkdir -p build && cd build
+
+$ ../configure
+
+$ make -j8
+# output omitted because it is very long
+
+$ make check
+# output omitted because it is very long
+PASS: testcases/complete-run.pl
+============================================================================
+Testsuite summary for i3 4.13
+============================================================================
+# TOTAL: 1
+# PASS: 1
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+============================================================================
+
+$ less test-suite.log
+----------------------------------------
+
+==== Coverage testing
+
+Coverage testing is possible with +lcov+, the front-end for GCC's coverage
+testing tool +gcov+. The testcases can generate a nice html report that tells
+you which functions and lines were covered during a run of the tests. You can
+use this tool to judge how effective your tests are.
+
+To use test coverage tools, first compile with coverage enabled.
+
+---------------------------------------------------
+COVERAGE=1 make
+---------------------------------------------------
+
+Then run the tests with the +--coverage-testing+ flag.
+
+---------------------------------------------------
+./complete-run.pl --coverage-testing
+---------------------------------------------------
+
+Then open +latest/i3-coverage/index.html+ in your web browser.
+
==== IPC interface
The testsuite makes extensive use of the IPC (Inter-Process Communication)
terminating it cleanly and (most importantly) for modifying and getting the
current state (layout tree).
-See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
+See [https://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
==== X11::XCB
testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl
module which uses the XCB protocol description to generate Perl bindings to
X11. They work in a very similar way to libxcb (which i3 uses) and provide
-relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as
+relatively high-level interfaces (objects such as +X11::XCB::Window+) as well as
access to the low-level interface, which is very useful when testing a window
manager.
=== Filesystem structure
In the git root of i3, the testcases live in the folder +testcases+. This
-folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
-configuration file which will be used for the tests. The different testcases
-(their file extension is .t, not .pl) themselves can be found in the
-conventionally named subfolder +t+:
+folder contains the +complete-run.pl+ and a base configuration file which will
+be used for the tests. The different testcases (their file extension is .t, not
+.pl) themselves can be found in the conventionally named subfolder +t+:
.Filesystem structure
--------------------------------------------
├── testcases
│ ├── complete-run.pl
│ ├── i3-test.config
+│ ├── lib
+│ │ ├── i3test.pm
+│ │ ├── SocketActivation.pm
+│ │ └── StartXDummy.pm
│ ├── t
│ │ ├── 00-load.t
│ │ ├── 01-tile.t
│ │ ├── ...
│ │ ├── omitted for brevity
│ │ ├── ...
-│ │ ├── 74-regress-focus-toggle.t
-│ │ └── lib
-│ │ └── i3test.pm
-│ └── Xdummy
+│ │ └── 74-regress-focus-toggle.t
--------------------------------------------
== Anatomy of a testcase
is($x->input_focus, $left->id, 'left window focused');
----------
-However, the test fails. Sometimes. Apparantly, there is a race condition in
+However, the test fails. Sometimes. Apparently, there is a race condition in
your test. If you think about it, this is because you are using two different
pieces of software: You tell i3 to update focus, i3 confirms that, and then you
ask X11 to give you the current focus. There is a certain time i3 needs to
it significantly more attractive to run the test suite more often (or at all)
during development.
-An alternative approach to using socket activation is polling for the existance
+An alternative approach to using socket activation is polling for the existence
of the IPC socket and connecting to it. While this might be slightly easier to
-implement, it wastes CPU time and is considerably more ugly than this solution
+implement, it wastes CPU time and is considerably uglier than this solution
:). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.
projects / i3 / i3 / blobdiff