i3 testsuite
============
-Michael Stapelberg <michael+i3@stapelberg.de>
-September 2011
+Michael Stapelberg <michael@i3wm.org>
+September 2012
-This document explains how the i3 testsuite works, how to use it and how to
-extend it. It is targeted at developers who not necessarily have been doing
-testing before or have not been testing in Perl before. In general, the
-testsuite is not of interest for end users.
+This document explains how the i3 testsuite works, how to use it and extend it.
+It is targeted at developers who not necessarily have been doing testing before
+or have not been testing in Perl before. In general, the testsuite is not of
+interest for end users.
== Introduction
The goal of having these tests is to automatically find problems and to
automatically get a feel for whether a change in the source code breaks any
existing feature. After every modification of the i3 sourcecode, the developer
-should run the full testsuite. If one of the tests does not pass (but fails),
-the corresponding problem should be fixed (or, in some cases, the testcase has
-to be modified). For every bugreport, a testcase should be written to test the
-correct behaviour. Initially, it will fail, but after fixing the bug, it will
-pass. This ensures (or increases the chance) that bugs which have been fixed
-once will never be found again.
+should run the full testsuite. If one of the tests fails, the corresponding
+problem should be fixed (or, in some cases, the testcase has to be modified).
+For every bugreport, a testcase should be written to test the correct
+behaviour. Initially, it will fail, but after fixing the bug, it will pass.
+This ensures (or increases the chance) that bugs which have been fixed once
+will never be found again.
Also, when implementing a new feature, a testcase might be a good way to be
able to easily test if the feature is working correctly. Many developers will
test manually if everything works. Having a testcase not only helps you with
that, but it will also be useful for every future change.
+== Relevant documentation
+
+Apart from this document, you should also have a look at:
+
+1. The "Modern Perl" book, which can be found at
+ http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf
+2. The latest Perl documentation of the "i3test" (general testcase setup) and
+ "i3test::Test" (additional test instructions) modules:
+ https://build.i3wm.org/docs/lib-i3test.html respectively
+ https://build.i3wm.org/docs/lib-i3test-test.html
+3. The latest documentation on i3’s IPC interface:
+ https://build.i3wm.org/docs/ipc.html
+
== Implementation
For several reasons, the i3 testsuite has been implemented in Perl:
2. Perl is widely available and has a well-working package infrastructure.
3. The author is familiar with Perl :).
+4. It is a good idea to use a different language for the tests than the
+ implementation itself.
Please do not start programming language flamewars at this point.
+=== Installing the dependencies
+
+As usual with Perl programs, the testsuite ships with a +Makefile.PL+.
+This file specifies which Perl modules the testsuite depends on and can be used
+to install all of them.
+
+Perl modules are distributed via CPAN, and there is the official, standard CPAN
+client, simply called +cpan+. It comes with every Perl installation and can be
+used to install the testsuite. Many users prefer to use the more modern
++cpanminus+ instead, though (because it asks no questions and just works):
+
+The tests additionally require +Xephyr(1)+ to run a nested X server. Install
++xserver-xephyr+ on Debian or +xorg-server-xephyr+ on Arch Linux.
+
+.Installing testsuite dependencies using cpanminus (preferred)
+--------------------------------------------------------------------------------
+$ cd ~/i3/testcases
+$ sudo apt-get install cpanminus
+$ sudo cpanm .
+$ cd ~/i3/AnyEvent-I3
+$ sudo cpanm Module::Install
+$ sudo cpanm .
+--------------------------------------------------------------------------------
+
+If you don’t want to use cpanminus for some reason, the same works with cpan:
+
+.Installing testsuite dependencies using cpan
+--------------------------------------------------------------------------------
+$ cd ~/i3/testcases
+$ sudo cpan .
+$ cd ~/i3/AnyEvent-I3
+$ sudo cpan Module::Install
+$ sudo cpan .
+--------------------------------------------------------------------------------
+
+In case you don’t have root permissions, you can also install into your home
+directory, see https://michael.stapelberg.de/cpan/
+
=== Mechanisms
==== Script: complete-run
testcases by default, but you can be more specific and let it only run one or
more testcases. Also, it takes care of starting up a separate instance of i3
with an appropriate configuration file and creates a folder for each run
-containing the appropriate i3 logfile for each testcase. The latest folder can
-always be found under the symlink +latest/+. It is recommended that you run the
-tests on one or more separate X server instances (you can only start one window
-manager per X session), for example using the provided Xdummy script.
-+complete-run.pl+ takes one or more X11 display specifications and parallelizes
-the testcases appropriately:
-
-.Example invocation of complete-run.pl+
+containing the appropriate i3 logfile for each testcase. The latest folder can
+always be found under the symlink +latest/+. Unless told differently, it will
+run the tests on a separate X server instance (using Xephyr).
+
+Xephyr will open a window where you can inspect the running test. By default,
+tests are run under Xvfb.
+
+.Example invocation of +complete-run.pl+
---------------------------------------
-$ cd ~/i3/testcases
+$ cd ~/i3
+
+$ autoreconf -fi
+
+$ mkdir -p build && cd build
-# start two dummy X11 instances in the background
-$ ./Xdummy :1 &
-$ ./Xdummy :2 &
+$ ../configure
-$ ./complete-run.pl -d :1,:2
+$ make -j8
+# output omitted because it is very long
+
+$ cd testcases
+
+$ ./complete-run.pl
# output omitted because it is very long
All tests successful.
Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU)
Result: PASS
-$ ./complete-run.pl -d :1 t/04-floating.t
+$ ./complete-run.pl t/04-floating.t
[:3] i3 startup: took 0.07s, status = 1
[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
[:3] t/04-floating.t finished
$ less latest/i3-log-for-04-floating.t
----------------------------------------
+If your attempt to run the tests with a bare call to ./complete-run.pl fails, try this:
+
+---------------------------------------------------
+$ ./complete-run.pl --parallel=1 --keep-xserver-output
+---------------------------------------------------
+
+This will show the output of Xephyr, which is the X server implementation we
+use for testing.
+
+===== make command: +make check+
+Make check runs the i3 testsuite.
+You can still use ./testcases/complete-run.pl to get the interactive progress output.
+
+.Example invocation of +make check+
+---------------------------------------
+$ cd ~/i3
+
+$ autoreconf -fi
+
+$ mkdir -p build && cd build
+
+$ ../configure
+
+$ make -j8
+# output omitted because it is very long
+
+$ make check
+# output omitted because it is very long
+PASS: testcases/complete-run.pl
+============================================================================
+Testsuite summary for i3 4.13
+============================================================================
+# TOTAL: 1
+# PASS: 1
+# SKIP: 0
+# XFAIL: 0
+# FAIL: 0
+# XPASS: 0
+# ERROR: 0
+============================================================================
+
+$ less test-suite.log
+----------------------------------------
+
+==== Coverage testing
+
+Coverage testing is possible with +lcov+, the front-end for GCC's coverage
+testing tool +gcov+. The testcases can generate a nice html report that tells
+you which functions and lines were covered during a run of the tests. You can
+use this tool to judge how effective your tests are.
+
+To use test coverage tools, first compile with coverage enabled.
+
+---------------------------------------------------
+COVERAGE=1 make
+---------------------------------------------------
+
+Then run the tests with the +--coverage-testing+ flag.
+
+---------------------------------------------------
+./complete-run.pl --coverage-testing
+---------------------------------------------------
+
+Then open +latest/i3-coverage/index.html+ in your web browser.
+
==== IPC interface
The testsuite makes extensive use of the IPC (Inter-Process Communication)
terminating it cleanly and (most importantly) for modifying and getting the
current state (layout tree).
-See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
+See [https://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
==== X11::XCB
testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl
module which uses the XCB protocol description to generate Perl bindings to
X11. They work in a very similar way to libxcb (which i3 uses) and provide
-relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as
+relatively high-level interfaces (objects such as +X11::XCB::Window+) as well as
access to the low-level interface, which is very useful when testing a window
manager.
=== Filesystem structure
In the git root of i3, the testcases live in the folder +testcases+. This
-folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
-configuration file which will be used for the tests. The different testcases
-themselve can be found in the conventionally named subfolder +t+:
+folder contains the +complete-run.pl+ and a base configuration file which will
+be used for the tests. The different testcases (their file extension is .t, not
+.pl) themselves can be found in the conventionally named subfolder +t+:
.Filesystem structure
--------------------------------------------
├── testcases
│ ├── complete-run.pl
│ ├── i3-test.config
+│ ├── lib
+│ │ ├── i3test.pm
+│ │ ├── SocketActivation.pm
+│ │ └── StartXDummy.pm
│ ├── t
│ │ ├── 00-load.t
│ │ ├── 01-tile.t
│ │ ├── ...
│ │ ├── omitted for brevity
│ │ ├── ...
-│ │ ├── 74-regress-focus-toggle.t
-│ │ └── lib
-│ │ └── i3test.pm
-│ └── Xdummy
+│ │ └── 74-regress-focus-toggle.t
--------------------------------------------
== Anatomy of a testcase
----------------------
#
# Returns the input focus after sending the given command to i3 via IPC
-# end sleeping for half a second to make sure i3 reacted
+# and syncing with i3
#
sub focus_after {
my $msg = shift;
is($focus, $mid->id, "focus unchanged");
----------------------
+Syntax hint: The qq keyword is the interpolating quote operator. It lets you
+chose a quote character (in this case the +|+ character, a pipe). This makes
+having double quotes in our string easy.
+
In this new major section, a random mark (mark is an identifier for a window,
see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we
test that trying to focus that mark will not do anything. This is important: Do
Remember: Focus was on the middle window (we verified that earlier in "Test
assumptions"). We now mark the middle window with our randomly generated mark.
Afterwards, we switch focus away from the middle window to be able to tell if
-focusing it via its mark will work. If it does work (next test), the goto
-command works.
+focusing it via its mark will work. If the test works, the goto command seems
+to be working.
.t/11-goto.t: Test corner case
----------------------
is($x->input_focus, $left->id, 'left window focused');
----------
-However, the test fails. Sometimes. Apparantly, there is a race condition in
+However, the test fails. Sometimes. Apparently, there is a race condition in
your test. If you think about it, this is because you are using two different
pieces of software: You tell i3 to update focus, i3 confirms that, and then you
ask X11 to give you the current focus. There is a certain time i3 needs to
the same one when getting the reply.
== Appendix B: Socket activation
+
+Socket activation is a mechanism which was made popular by systemd, an init
+replacement. It basically describes creating a listening socket before starting
+a program. systemd will invoke the program only when an actual connection to
+the socket is made, hence the term socket activation.
+
+The interesting part of this (in the i3 context) is that you can very precisely
+detect when the program is ready (finished its initialization).
+
+=== Preparing the listening socket
+
++complete-run.pl+ will create a listening UNIX socket which it will then pass
+to i3. This socket will be used by i3 as an additional IPC socket, just like
+the one it will create on its own. Passing the socket happens implicitly
+because children will inherit the parent’s sockets when fork()ing and sockets
+will continue to exist after an exec() call (unless CLOEXEC is set of course).
+
+The only explicit things +complete-run.pl+ has to do is setting the +LISTEN_FDS+
+environment variable to the number of sockets which exist (1 in our case) and
+setting the +LISTEN_PID+ environment variable to the current process ID. Both
+variables are necessary so that the program (i3) knows how many sockets it
+should use and if the environment variable is actually intended for it. i3 will
+then start looking for sockets at file descriptor 3 (since 0, 1 and 2 are used
+for stdin, stdout and stderr, respectively).
+
+The actual Perl code which sets up the socket, fork()s, makes sure the socket
+has file descriptor 3 and sets up the environment variables follows (shortened
+a bit):
+
+
+.Setup socket and environment
+-----------------------------
+my $socket = IO::Socket::UNIX->new(
+ Listen => 1,
+ Local => $args{unix_socket_path},
+);
+
+my $pid = fork;
+if ($pid == 0) {
+ $ENV{LISTEN_PID} = $$;
+ $ENV{LISTEN_FDS} = 1;
+
+ # Only pass file descriptors 0 (stdin), 1 (stdout),
+ # 2 (stderr) and 3 (socket) to the child.
+ $^F = 3;
+
+ # If the socket does not use file descriptor 3 by chance
+ # already, we close fd 3 and dup2() the socket to 3.
+ if (fileno($socket) != 3) {
+ POSIX::close(3);
+ POSIX::dup2(fileno($socket), 3);
+ }
+
+ exec "/usr/bin/i3";
+}
+-----------------------------
+
+=== Waiting for a reply
+
+In the parent process, we want to know when i3 is ready to answer our IPC
+requests and handle our windows. Therefore, after forking, we immediately close
+the listening socket (i3 will handle this side of the socket) and connect to it
+(remember, we are talking about a named UNIX socket) as a client. This connect
+call will immediately succeed because the kernel buffers it. Then, we send a
+request (of type GET_TREE, but that is not really relevant). Writing data to
+the socket will also succeed immediately because, again, the kernel buffers it
+(only up to a certain amount of data of course).
+
+Afterwards, we just blockingly wait until we get an answer. In the child
+process, i3 will setup the listening socket in its event loop. Immediately
+after actually starting the event loop, it will notice a new client connecting
+(the parent process) and handle its request. Since all initialization has been
+completed successfully by the time the event loop is entered, we can now assume
+that i3 is ready.
+
+=== Timing and conclusion
+
+A beautiful feature of this mechanism is that it does not depend on timing. It
+does not matter when the child process gets CPU time or when the parent process
+gets CPU time. On heavily loaded machines (or machines with multiple CPUs,
+cores or unreliable schedulers), this makes waiting for i3 much more robust.
+
+Before using socket activation, we typically used a +sleep(1)+ and hoped that
+i3 was initialized by that time. Of course, this breaks on some (slow)
+computers and wastes a lot of time on faster computers. By using socket
+activation, we decreased the total amount of time necessary to run all tests
+(72 files at the time of writing) from > 100 seconds to 16 seconds. This makes
+it significantly more attractive to run the test suite more often (or at all)
+during development.
+
+An alternative approach to using socket activation is polling for the existence
+of the IPC socket and connecting to it. While this might be slightly easier to
+implement, it wastes CPU time and is considerably uglier than this solution
+:). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.
projects / i3 / i3 / blobdiff