Michael Stapelberg <michael+i3@stapelberg.de>
September 2011
-This document explains how the i3 testsuite works, how to use it and how to
-extend it. It is targeted at developers who not necessarily have been doing
-testing before or have not been testing in Perl before. In general, the
-testsuite is not of interest for end users.
+This document explains how the i3 testsuite works, how to use it and extend it.
+It is targeted at developers who not necessarily have been doing testing before
+or have not been testing in Perl before. In general, the testsuite is not of
+interest for end users.
== Introduction
The goal of having these tests is to automatically find problems and to
automatically get a feel for whether a change in the source code breaks any
existing feature. After every modification of the i3 sourcecode, the developer
-should run the full testsuite. If one of the tests does not pass (but fails),
-the corresponding problem should be fixed (or, in some cases, the testcase has
-to be modified). For every bugreport, a testcase should be written to test the
-correct behaviour. Initially, it will fail, but after fixing the bug, it will
-pass. This ensures (or increases the chance) that bugs which have been fixed
-once will never be found again.
+should run the full testsuite. If one of the tests fails, the corresponding
+problem should be fixed (or, in some cases, the testcase has to be modified).
+For every bugreport, a testcase should be written to test the correct
+behaviour. Initially, it will fail, but after fixing the bug, it will pass.
+This ensures (or increases the chance) that bugs which have been fixed once
+will never be found again.
Also, when implementing a new feature, a testcase might be a good way to be
able to easily test if the feature is working correctly. Many developers will
2. Perl is widely available and has a well-working package infrastructure.
3. The author is familiar with Perl :).
+4. It is a good idea to use a different language for the tests than the
+ implementation itself.
Please do not start programming language flamewars at this point.
testcases by default, but you can be more specific and let it only run one or
more testcases. Also, it takes care of starting up a separate instance of i3
with an appropriate configuration file and creates a folder for each run
-containing the appropriate i3 logfile for each testcase. The latest folder can
-always be found under the symlink +latest/+. It is recommended that you run the
-tests on one or more separate X server instances (you can only start one window
-manager per X session), for example using the provided Xdummy script.
-+complete-run.pl+ takes one or more X11 display specifications and parallelizes
-the testcases appropriately:
+containing the appropriate i3 logfile for each testcase. The latest folder can
+always be found under the symlink +latest/+. Unless told differently, it will
+run the tests on a separate X server instance (using the Xdummy script).
.Example invocation of complete-run.pl+
---------------------------------------
$ cd ~/i3/testcases
-# start two dummy X11 instances in the background
-$ ./Xdummy :1 &
-$ ./Xdummy :2 &
-
-$ ./complete-run.pl -d :1,:2
+$ ./complete-run.pl
# output omitted because it is very long
All tests successful.
Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU)
Result: PASS
-$ ./complete-run.pl -d :1 t/04-floating.t
+$ ./complete-run.pl t/04-floating.t
[:3] i3 startup: took 0.07s, status = 1
[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
[:3] t/04-floating.t finished
In the git root of i3, the testcases live in the folder +testcases+. This
folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
configuration file which will be used for the tests. The different testcases
-themselve can be found in the conventionally named subfolder +t+:
+(their file extension is .t, not .pl) themselves can be found in the
+conventionally named subfolder +t+:
.Filesystem structure
--------------------------------------------
├── testcases
│ ├── complete-run.pl
│ ├── i3-test.config
+│ ├── lib
+│ │ ├── i3test.pm
+│ │ ├── SocketActivation.pm
+│ │ └── StartXDummy.pm
│ ├── t
│ │ ├── 00-load.t
│ │ ├── 01-tile.t
│ │ ├── ...
│ │ ├── omitted for brevity
│ │ ├── ...
-│ │ ├── 74-regress-focus-toggle.t
-│ │ └── lib
-│ │ └── i3test.pm
+│ │ └── 74-regress-focus-toggle.t
│ └── Xdummy
--------------------------------------------
----------------------
#
# Returns the input focus after sending the given command to i3 via IPC
-# end sleeping for half a second to make sure i3 reacted
+# and syncing with i3
#
sub focus_after {
my $msg = shift;
is($focus, $mid->id, "focus unchanged");
----------------------
+Syntax hint: The qq keyword is the interpolating quote operator. It lets you
+chose a quote character (in this case the +|+ character, a pipe). This makes
+having double quotes in our string easy.
+
In this new major section, a random mark (mark is an identifier for a window,
see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we
test that trying to focus that mark will not do anything. This is important: Do
Remember: Focus was on the middle window (we verified that earlier in "Test
assumptions"). We now mark the middle window with our randomly generated mark.
Afterwards, we switch focus away from the middle window to be able to tell if
-focusing it via its mark will work. If it does work (next test), the goto
-command works.
+focusing it via its mark will work. If the test works, the goto command seems
+to be working.
.t/11-goto.t: Test corner case
----------------------
the same one when getting the reply.
== Appendix B: Socket activation
+
+Socket activation is a mechanism which was made popular by systemd, an init
+replacement. It basically describes creating a listening socket before starting
+a program. systemd will invoke the program only when an actual connection to
+the socket is made, hence the term socket activation.
+
+The interesting part of this (in the i3 context) is that you can very precisely
+detect when the program is ready (finished its initialization).
+
+=== Preparing the listening socket
+
++complete-run.pl+ will create a listening UNIX socket which it will then pass
+to i3. This socket will be used by i3 as an additional IPC socket, just like
+the one it will create on its own. Passing the socket happens implicitly
+because children will inherit the parent’s sockets when fork()ing and sockets
+will continue to exist after an exec() call (unless CLOEXEC is set of course).
+
+The only explicit things +complete-run.pl+ has to do is setting the +LISTEN_FDS+
+environment variable to the number of sockets which exist (1 in our case) and
+setting the +LISTEN_PID+ environment variable to the current process ID. Both
+variables are necessary so that the program (i3) knows how many sockets it
+should use and if the environment variable is actually intended for it. i3 will
+then start looking for sockets at file descriptor 3 (since 0, 1 and 2 are used
+for stdin, stdout and stderr, respectively).
+
+The actual Perl code which sets up the socket, fork()s, makes sure the socket
+has file descriptor 3 and sets up the environment variables follows (shortened
+a bit):
+
+
+.Setup socket and environment
+-----------------------------
+my $socket = IO::Socket::UNIX->new(
+ Listen => 1,
+ Local => $args{unix_socket_path},
+);
+
+my $pid = fork;
+if ($pid == 0) {
+ $ENV{LISTEN_PID} = $$;
+ $ENV{LISTEN_FDS} = 1;
+
+ # Only pass file descriptors 0 (stdin), 1 (stdout),
+ # 2 (stderr) and 3 (socket) to the child.
+ $^F = 3;
+
+ # If the socket does not use file descriptor 3 by chance
+ # already, we close fd 3 and dup2() the socket to 3.
+ if (fileno($socket) != 3) {
+ POSIX::close(3);
+ POSIX::dup2(fileno($socket), 3);
+ }
+
+ exec "/usr/bin/i3";
+}
+-----------------------------
+
+=== Waiting for a reply
+
+In the parent process, we want to know when i3 is ready to answer our IPC
+requests and handle our windows. Therefore, after forking, we immediately close
+the listening socket (i3 will handle this side of the socket) and connect to it
+(remember, we are talking about a named UNIX socket) as a client. This connect
+call will immediately succeed because the kernel buffers it. Then, we send a
+request (of type GET_TREE, but that is not really relevant). Writing data to
+the socket will also succeed immediately because, again, the kernel buffers it
+(only up to a certain amount of data of course).
+
+Afterwards, we just blockingly wait until we get an answer. In the child
+process, i3 will setup the listening socket in its event loop. Immediately
+after actually starting the event loop, it will notice a new client connecting
+(the parent process) and handle its request. Since all initialization has been
+completed successfully by the time the event loop is entered, we can now assume
+that i3 is ready.
+
+=== Timing and conclusion
+
+A beautiful feature of this mechanism is that it does not depend on timing. It
+does not matter when the child process gets CPU time or when the parent process
+gets CPU time. On heavily loaded machines (or machines with multiple CPUs,
+cores or unreliable schedulers), this makes waiting for i3 much more robust.
+
+Before using socket activation, we typically used a +sleep(1)+ and hoped that
+i3 was initialized by that time. Of course, this breaks on some (slow)
+computers and wastes a lot of time on faster computers. By using socket
+activation, we decreased the total amount of time necessary to run all tests
+(72 files at the time of writing) from > 100 seconds to 16 seconds. This makes
+it significantly more attractive to run the test suite more often (or at all)
+during development.
+
+An alternative approach to using socket activation is polling for the existance
+of the IPC socket and connecting to it. While this might be slightly easier to
+implement, it wastes CPU time and is considerably uglier than this solution
+:). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.
projects / i3 / i3 / blobdiff