3 Michael Stapelberg <michael@i3wm.org>
6 This document explains how the i3 testsuite works, how to use it and extend it.
7 It is targeted at developers who not necessarily have been doing testing before
8 or have not been testing in Perl before. In general, the testsuite is not of
9 interest for end users.
14 The i3 testsuite is a collection of files which contain testcases for various
15 i3 features. Some of them test if a certain workflow works correctly (moving
16 windows, focus behaviour, …). Others are regression tests and contain code
17 which previously made i3 crash or lead to unexpected behaviour. They then check
18 if i3 still runs (meaning it did not crash) and if it handled everything
21 The goal of having these tests is to automatically find problems and to
22 automatically get a feel for whether a change in the source code breaks any
23 existing feature. After every modification of the i3 sourcecode, the developer
24 should run the full testsuite. If one of the tests fails, the corresponding
25 problem should be fixed (or, in some cases, the testcase has to be modified).
26 For every bugreport, a testcase should be written to test the correct
27 behaviour. Initially, it will fail, but after fixing the bug, it will pass.
28 This ensures (or increases the chance) that bugs which have been fixed once
29 will never be found again.
31 Also, when implementing a new feature, a testcase might be a good way to be
32 able to easily test if the feature is working correctly. Many developers will
33 test manually if everything works. Having a testcase not only helps you with
34 that, but it will also be useful for every future change.
36 == Relevant documentation
38 Apart from this document, you should also have a look at:
40 1. The "Modern Perl" book, which can be found at
41 http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf
42 2. The latest Perl documentation of the "i3test" (general testcase setup) and
43 "i3test::Test" (additional test instructions) modules:
44 http://build.i3wm.org/docs/lib-i3test.html respectively
45 http://build.i3wm.org/docs/lib-i3test-test.html
46 3. The latest documentation on i3’s IPC interface:
47 http://build.i3wm.org/docs/ipc.html
51 For several reasons, the i3 testsuite has been implemented in Perl:
53 1. Perl has a long tradition of testing. Every popular/bigger Perl module which
54 you can find on CPAN will not only come with documentation, but also with
55 tests. Therefore, the available infrastructure for tests is comprehensive.
56 See for example the excellent http://search.cpan.org/perldoc?Test::More
57 and the referenced http://search.cpan.org/perldoc?Test::Tutorial.
59 2. Perl is widely available and has a well-working package infrastructure.
60 3. The author is familiar with Perl :).
61 4. It is a good idea to use a different language for the tests than the
62 implementation itself.
64 Please do not start programming language flamewars at this point.
66 === Installing the dependencies
68 As usual with Perl programs, the testsuite ships with a +Makefile.PL+.
69 This file specifies which Perl modules the testsuite depends on and can be used
70 to install all of them.
72 Perl modules are distributed via CPAN, and there is the official, standard CPAN
73 client, simply called +cpan+. It comes with every Perl installation and can be
74 used to install the testsuite. Many users prefer to use the more modern
75 +cpanminus+ instead, though (because it asks no questions and just works):
77 .Installing testsuite dependencies using cpanminus (preferred)
78 --------------------------------------------------------------------------------
80 $ sudo apt-get install cpanminus
82 --------------------------------------------------------------------------------
84 If you don’t want to use cpanminus for some reason, the same works with cpan:
86 .Installing testsuite dependencies using cpan
87 --------------------------------------------------------------------------------
90 --------------------------------------------------------------------------------
92 In case you don’t have root permissions, you can also install into your home
93 directory, see http://michael.stapelberg.de/cpan/
97 ==== Script: complete-run
99 The testcases are run by a script called +complete-run.pl+. It runs all
100 testcases by default, but you can be more specific and let it only run one or
101 more testcases. Also, it takes care of starting up a separate instance of i3
102 with an appropriate configuration file and creates a folder for each run
103 containing the appropriate i3 logfile for each testcase. The latest folder can
104 always be found under the symlink +latest/+. Unless told differently, it will
105 run the tests on a separate X server instance (using the Xdummy script).
107 .Example invocation of complete-run.pl+
108 ---------------------------------------
112 # output omitted because it is very long
113 All tests successful.
114 Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU)
117 $ ./complete-run.pl t/04-floating.t
118 [:3] i3 startup: took 0.07s, status = 1
119 [:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
120 [:3] t/04-floating.t finished
122 output for t/04-floating.t:
123 ok 1 - use X11::XCB::Window;
124 ok 2 - The object isa X11::XCB::Window
125 ok 3 - Window is mapped
126 ok 4 - i3 raised the width to 75
127 ok 5 - i3 raised the height to 50
128 ok 6 - i3 did not map it to (0x0)
129 ok 7 - The object isa X11::XCB::Window
130 ok 8 - i3 let the width at 80
131 ok 9 - i3 let the height at 90
132 ok 10 - i3 mapped it to x=1
133 ok 11 - i3 mapped it to y=18
134 ok 12 - The object isa X11::XCB::Window
135 ok 13 - i3 let the width at 80
136 ok 14 - i3 let the height at 90
139 All tests successful.
140 Files=1, Tests=14, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.19 cusr 0.03 csys = 0.23 CPU)
143 $ less latest/i3-log-for-04-floating.t
144 ----------------------------------------
148 The testsuite makes extensive use of the IPC (Inter-Process Communication)
149 interface which i3 provides. It is used for the startup process of i3, for
150 terminating it cleanly and (most importantly) for modifying and getting the
151 current state (layout tree).
153 See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
157 In order to open new windows, change attributes, get events, etc., the
158 testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl
159 module which uses the XCB protocol description to generate Perl bindings to
160 X11. They work in a very similar way to libxcb (which i3 uses) and provide
161 relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as
162 access to the low-level interface, which is very useful when testing a window
165 === Filesystem structure
167 In the git root of i3, the testcases live in the folder +testcases+. This
168 folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
169 configuration file which will be used for the tests. The different testcases
170 (their file extension is .t, not .pl) themselves can be found in the
171 conventionally named subfolder +t+:
173 .Filesystem structure
174 --------------------------------------------
176 │ ├── complete-run.pl
180 │ │ ├── SocketActivation.pm
181 │ │ └── StartXDummy.pm
185 │ │ ├── 02-fullscreen.t
187 │ │ ├── omitted for brevity
189 │ │ └── 74-regress-focus-toggle.t
191 --------------------------------------------
193 == Anatomy of a testcase
195 Learning by example is definitely a good strategy when you are wondering how to
196 write a testcase. Let's take +t/11-goto.t+ as an easy example and go through it
199 .t/11-goto.t: Boilerplate
200 ----------------------
202 # vim:ts=4:sw=4:expandtab
207 my $x = X11::XCB::Connection->new;
208 -----------------------
210 This is what we call boilerplate. It exists at the top of every test file (to
211 some extent). The first line is the shebang, which specifies that this file is
212 a Perl script. The second line contains VIM specific settings on how to
213 edit/format this file (use spaces instead of tabs, indent using 4 spaces).
214 Afterwards, the +i3test+ module is used. This module contains i3 testsuite
215 specific functions which you are strongly encouraged to use. They make writing
216 testcases a lot easier and will make it easier for other people to read your
219 The next line uses the +File::Temp+ module. This is specific to this testcase,
220 because it needs to generate a temporary name during the test. Many testcases
221 use only the +i3test+ module.
223 The last line opens a connection to X11. You might or might not need this in
224 your testcase, depending on whether you are going to open windows (etc.) or
225 only use i3 commands.
228 ----------------------
229 my $tmp = fresh_workspace;
232 ----------------------
234 The first line calls i3test's +fresh_workspace+ function which looks for a
235 currently unused workspace, switches to it, and returns its name. The variable
236 +$tmp+ will end up having a value such as +"/tmp/87kBVcHbA9"+. Note that this
237 is not (necessarily) a valid path, it's just a random workspace name.
239 So, now that we are on a new workspace, we ensure that the workspace uses
240 horizontal orientation by issuing the +split h+ command (see the i3 User's
241 Guide for a list of commands). This is not strictly necessary, but good style.
242 In general, the +cmd+ function executes the specified i3 command by using the
243 IPC interface and returns once i3 acknowledged the command.
246 ----------------------
247 #####################################################################
248 # Create two windows and make sure focus switching works
249 #####################################################################
251 my $top = open_window($x);
252 my $mid = open_window($x);
253 my $bottom = open_window($x);
254 ----------------------
256 In every major section of a testcase, you should put a comment like the one
257 above. This makes it immediately clear how the file is structured.
259 The +open_window+ function opens a standard window, which will then be put into
260 tiling mode by i3. If you want a floating window, use the
261 +open_floating_window+ function. These functions accept the same parameters as
262 +X11::XCB::Window->new+, see the i3test documentation at TODO.
264 .t/11-goto.t: Helper function
265 ----------------------
267 # Returns the input focus after sending the given command to i3 via IPC
268 # and syncing with i3
275 return $x->input_focus;
277 ----------------------
279 This section defines a helper function which will be used over and over in this
280 testcase. If you have code which gets executed more than once or twice
281 (depending on the length of your test, use your best judgement), please put it
282 in a function. Tests should be short, concise and clear.
284 The +focus_after+ function executes a command and returns the X11 focus after
285 the command was executed. The +sync_with_i3+ command makes sure that i3 could
286 push its state to X11. See <<i3_sync>> to learn how this works exactly.
288 .t/11-goto.t: Test assumptions
289 ----------------------
290 $focus = $x->input_focus;
291 is($focus, $bottom->id, "Latest window focused");
293 $focus = focus_after('focus left');
294 is($focus, $mid->id, "Middle window focused");
295 ----------------------
297 Now, we run the first two real tests. They use +Test::More+'s +is+ function,
298 which compares two values and prints the differences if they are not the same.
299 After the arguments, we supply a short comment to indicate what we are testing
300 here. This makes it vastly more easy for the developer to spot which testcase
301 is the problem in case one fails.
303 The first test checks that the most recently opened window is focused.
304 Afterwards, the command +focus left+ is issued and it is verified that the
305 middle window now has focus.
307 Note that this is not a comprehensive test of the +focus+ command -- we would
308 have to test wrapping, focus when using a more complex layout, focusing the
309 parent/child containers, etc. But that is not the point of this testcase.
310 Instead, we just want to know if +$x->input_focus+ corresponds with what we are
311 expecting. If not, something is completely wrong with the test environment and
312 this trivial test will fail.
314 .t/11-goto.t: Test that the feature does not work (yet)
315 ----------------------
316 #####################################################################
317 # Now goto a mark which does not exist
318 #####################################################################
320 my $random_mark = mktemp('mark.XXXXXX');
322 $focus = focus_after(qq|[con_mark="$random_mark"] focus|);
323 is($focus, $mid->id, "focus unchanged");
324 ----------------------
326 Syntax hint: The qq keyword is the interpolating quote operator. It lets you
327 chose a quote character (in this case the +|+ character, a pipe). This makes
328 having double quotes in our string easy.
330 In this new major section, a random mark (mark is an identifier for a window,
331 see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we
332 test that trying to focus that mark will not do anything. This is important: Do
333 not only test that using a feature has the expected outcome, but also test that
334 using it without properly initializing it does no harm. This command could for
335 example have changed focus anyways (a bug) or crash i3 (obviously a bug).
337 .t/11-goto.t: Test that the feature does work
338 ----------------------
339 cmd "mark $random_mark";
341 $focus = focus_after('focus left');
342 is($focus, $top->id, "Top window focused");
344 $focus = focus_after(qq|[con_mark="$random_mark"] focus|);
345 is($focus, $mid->id, "goto worked");
346 ----------------------
348 Remember: Focus was on the middle window (we verified that earlier in "Test
349 assumptions"). We now mark the middle window with our randomly generated mark.
350 Afterwards, we switch focus away from the middle window to be able to tell if
351 focusing it via its mark will work. If the test works, the goto command seems
354 .t/11-goto.t: Test corner case
355 ----------------------
356 # check that we can specify multiple criteria
358 $focus = focus_after('focus left');
359 is($focus, $top->id, "Top window focused");
361 $focus = focus_after(qq|[con_mark="$random_mark" con_mark="$random_mark"] focus|);
362 is($focus, $mid->id, "goto worked");
363 ----------------------
365 Now we test the same feature, but specifying the mark twice in the command.
366 This should have no effect, but let’s be sure: test it and see if things go
369 .t/11-goto.t: Test second code path
370 ----------------------
371 #####################################################################
372 # Check whether the focus command will switch to a different
373 # workspace if necessary
374 #####################################################################
376 my $tmp2 = fresh_workspace;
378 is(focused_ws(), $tmp2, 'tmp2 now focused');
380 cmd qq|[con_mark="$random_mark"] focus|;
382 is(focused_ws(), $tmp, 'tmp now focused');
383 ----------------------
385 This part of the test checks that focusing windows by mark works across
386 workspaces. It uses i3test's +focused_ws+ function to get the current
389 .t/11-goto.t: Test second code path
390 ----------------------
392 ----------------------
394 The end of every testcase has to contain the +done_testing+ line. This tells
395 +complete-run.pl+ that the test was finished successfully. If it does not
396 occur, the test might have crashed during execution -- some of the reasons why
397 that could happen are bugs in the used modules, bugs in the testcase itself or
398 an i3 crash resulting in the testcase being unable to communicate with i3 via
402 == Appendix A: The i3 sync protocol
404 Consider the following situation: You open two windows in your testcase, then
405 you use +focus left+ and want to verify that the X11 focus has been updated
406 properly. Sounds simple, right? Let’s assume you use this straight-forward
409 .Racey focus testcase
411 my $left = open_window($x);
412 my $right = open_window($x);
414 is($x->input_focus, $left->id, 'left window focused');
417 However, the test fails. Sometimes. Apparantly, there is a race condition in
418 your test. If you think about it, this is because you are using two different
419 pieces of software: You tell i3 to update focus, i3 confirms that, and then you
420 ask X11 to give you the current focus. There is a certain time i3 needs to
421 update the X11 state. If the testcase gets CPU time before X11 processed i3's
422 requests, the test will fail.
424 image::i3-sync.png["Diagram of the race condition", title="Diagram of the race condition"]
426 One way to "solve" this would be to add +sleep 0.5;+ after the +cmd+ call.
427 After 0.5 seconds it should be safe to assume that focus has been updated,
430 In practice, this usually works. However, it has several problems:
432 1. This is obviously not a clean solution, but a workaround. Ugly.
433 2. On very slow machines, this might not work. Unlikely, but in different
434 situations (a delay to wait for i3 to startup) the necessary time is much
435 harder to guess, even for fast machines.
436 3. This *wastes a lot of time*. Usually, your computer is much faster than 0.5s
437 to update the status. However, sometimes, it might take 0.4s, so we can’t
440 To illustrate how grave the problem with wasting time actually is: Before
441 removing all sleeps from the testsuite, a typical run using 4 separate X
442 servers took around 50 seconds on my machine. After removing all the sleeps,
443 we achieved times of about 25 seconds. This is very significant and influences
444 the way you think about tests -- the faster they are, the more likely you are
445 to check whether everything still works quite often (which you should).
447 What I am trying to say is: Delays adds up quickly and make the test suite
450 The real solution for this problem is a mechanism which I call "the i3 sync
451 protocol". The idea is to send a request (which does not modify state) via X11
452 to i3 which will then be answered. Due to the request's position in the event
453 queue (*after* all previous events), you can be sure that by the time you
454 receive the reply, all other events have been dealt with by i3 (and, more
457 image::i3-sync-working.png["Diagram of the i3 sync solution", title="Diagram of the i3 sync solution"]
459 === Implementation details
461 The client which wants to sync with i3 initiates the protocol by sending a
462 ClientMessage to the X11 root window:
466 # Generate a ClientMessage, see xcb_client_message_t
467 my $msg = pack "CCSLLLLLLL",
468 CLIENT_MESSAGE, # response_type
471 $root, # destination window
472 $x->atom(name => 'I3_SYNC')->id,
474 $_sync_window->id, # data[0]: our own window id
475 $myrnd, # data[1]: a random value to identify the request
480 # Send it to the root window -- since i3 uses the SubstructureRedirect
481 # event mask, it will get the ClientMessage.
482 $x->send_event(0, $root, EVENT_MASK_SUBSTRUCTURE_REDIRECT, $msg);
485 i3 will then reply with the same ClientMessage, sent to the window specified in
486 +data[0]+. In the reply, +data[0]+ and +data[1]+ are exactly the same as in the
487 request. You should use a random value in +data[1]+ and check that you received
488 the same one when getting the reply.
490 == Appendix B: Socket activation
492 Socket activation is a mechanism which was made popular by systemd, an init
493 replacement. It basically describes creating a listening socket before starting
494 a program. systemd will invoke the program only when an actual connection to
495 the socket is made, hence the term socket activation.
497 The interesting part of this (in the i3 context) is that you can very precisely
498 detect when the program is ready (finished its initialization).
500 === Preparing the listening socket
502 +complete-run.pl+ will create a listening UNIX socket which it will then pass
503 to i3. This socket will be used by i3 as an additional IPC socket, just like
504 the one it will create on its own. Passing the socket happens implicitly
505 because children will inherit the parent’s sockets when fork()ing and sockets
506 will continue to exist after an exec() call (unless CLOEXEC is set of course).
508 The only explicit things +complete-run.pl+ has to do is setting the +LISTEN_FDS+
509 environment variable to the number of sockets which exist (1 in our case) and
510 setting the +LISTEN_PID+ environment variable to the current process ID. Both
511 variables are necessary so that the program (i3) knows how many sockets it
512 should use and if the environment variable is actually intended for it. i3 will
513 then start looking for sockets at file descriptor 3 (since 0, 1 and 2 are used
514 for stdin, stdout and stderr, respectively).
516 The actual Perl code which sets up the socket, fork()s, makes sure the socket
517 has file descriptor 3 and sets up the environment variables follows (shortened
521 .Setup socket and environment
522 -----------------------------
523 my $socket = IO::Socket::UNIX->new(
525 Local => $args{unix_socket_path},
530 $ENV{LISTEN_PID} = $$;
531 $ENV{LISTEN_FDS} = 1;
533 # Only pass file descriptors 0 (stdin), 1 (stdout),
534 # 2 (stderr) and 3 (socket) to the child.
537 # If the socket does not use file descriptor 3 by chance
538 # already, we close fd 3 and dup2() the socket to 3.
539 if (fileno($socket) != 3) {
541 POSIX::dup2(fileno($socket), 3);
546 -----------------------------
548 === Waiting for a reply
550 In the parent process, we want to know when i3 is ready to answer our IPC
551 requests and handle our windows. Therefore, after forking, we immediately close
552 the listening socket (i3 will handle this side of the socket) and connect to it
553 (remember, we are talking about a named UNIX socket) as a client. This connect
554 call will immediately succeed because the kernel buffers it. Then, we send a
555 request (of type GET_TREE, but that is not really relevant). Writing data to
556 the socket will also succeed immediately because, again, the kernel buffers it
557 (only up to a certain amount of data of course).
559 Afterwards, we just blockingly wait until we get an answer. In the child
560 process, i3 will setup the listening socket in its event loop. Immediately
561 after actually starting the event loop, it will notice a new client connecting
562 (the parent process) and handle its request. Since all initialization has been
563 completed successfully by the time the event loop is entered, we can now assume
566 === Timing and conclusion
568 A beautiful feature of this mechanism is that it does not depend on timing. It
569 does not matter when the child process gets CPU time or when the parent process
570 gets CPU time. On heavily loaded machines (or machines with multiple CPUs,
571 cores or unreliable schedulers), this makes waiting for i3 much more robust.
573 Before using socket activation, we typically used a +sleep(1)+ and hoped that
574 i3 was initialized by that time. Of course, this breaks on some (slow)
575 computers and wastes a lot of time on faster computers. By using socket
576 activation, we decreased the total amount of time necessary to run all tests
577 (72 files at the time of writing) from > 100 seconds to 16 seconds. This makes
578 it significantly more attractive to run the test suite more often (or at all)
581 An alternative approach to using socket activation is polling for the existance
582 of the IPC socket and connecting to it. While this might be slightly easier to
583 implement, it wastes CPU time and is considerably uglier than this solution
584 :). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.