X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=docs%2Ftestsuite;h=b535e7c141ca994b9f453771097868e9150c8104;hb=65eb54c0ba73dea6d01a2cb42c3c9b40df8c6820;hp=b3b76c745454879217e0351a8fbe7718d780b2d3;hpb=3537f2d4ca6ef2447a92f222d3458d4e27899661;p=i3%2Fi3 diff --git a/docs/testsuite b/docs/testsuite index b3b76c74..b535e7c1 100644 --- a/docs/testsuite +++ b/docs/testsuite @@ -1,7 +1,7 @@ i3 testsuite ============ -Michael Stapelberg -September 2011 +Michael Stapelberg +September 2012 This document explains how the i3 testsuite works, how to use it and extend it. It is targeted at developers who not necessarily have been doing testing before @@ -33,6 +33,19 @@ able to easily test if the feature is working correctly. Many developers will test manually if everything works. Having a testcase not only helps you with that, but it will also be useful for every future change. +== Relevant documentation + +Apart from this document, you should also have a look at: + +1. The "Modern Perl" book, which can be found at + http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf +2. The latest Perl documentation of the "i3test" (general testcase setup) and + "i3test::Test" (additional test instructions) modules: + https://build.i3wm.org/docs/lib-i3test.html respectively + https://build.i3wm.org/docs/lib-i3test-test.html +3. The latest documentation on i3’s IPC interface: + https://build.i3wm.org/docs/ipc.html + == Implementation For several reasons, the i3 testsuite has been implemented in Perl: @@ -45,9 +58,49 @@ For several reasons, the i3 testsuite has been implemented in Perl: 2. Perl is widely available and has a well-working package infrastructure. 3. The author is familiar with Perl :). +4. It is a good idea to use a different language for the tests than the + implementation itself. Please do not start programming language flamewars at this point. +=== Installing the dependencies + +As usual with Perl programs, the testsuite ships with a +Makefile.PL+. +This file specifies which Perl modules the testsuite depends on and can be used +to install all of them. + +Perl modules are distributed via CPAN, and there is the official, standard CPAN +client, simply called +cpan+. It comes with every Perl installation and can be +used to install the testsuite. Many users prefer to use the more modern ++cpanminus+ instead, though (because it asks no questions and just works): + +The tests additionally require +Xephyr(1)+ to run a nested X server. Install ++xserver-xephyr+ on Debian or +xorg-server-xephyr+ on Arch Linux. + +.Installing testsuite dependencies using cpanminus (preferred) +-------------------------------------------------------------------------------- +$ cd ~/i3/testcases +$ sudo apt-get install cpanminus +$ sudo cpanm . +$ cd ~/i3/AnyEvent-I3 +$ sudo cpanm Module::Install +$ sudo cpanm . +-------------------------------------------------------------------------------- + +If you don’t want to use cpanminus for some reason, the same works with cpan: + +.Installing testsuite dependencies using cpan +-------------------------------------------------------------------------------- +$ cd ~/i3/testcases +$ sudo cpan . +$ cd ~/i3/AnyEvent-I3 +$ sudo cpan Module::Install +$ sudo cpan . +-------------------------------------------------------------------------------- + +In case you don’t have root permissions, you can also install into your home +directory, see https://michael.stapelberg.de/cpan/ + === Mechanisms ==== Script: complete-run @@ -56,28 +109,35 @@ The testcases are run by a script called +complete-run.pl+. It runs all testcases by default, but you can be more specific and let it only run one or more testcases. Also, it takes care of starting up a separate instance of i3 with an appropriate configuration file and creates a folder for each run -containing the appropriate i3 logfile for each testcase. The latest folder can -always be found under the symlink +latest/+. It is recommended that you run the -tests on one or more separate X server instances (you can only start one window -manager per X session), for example using the provided Xdummy script. -+complete-run.pl+ takes one or more X11 display specifications and parallelizes -the testcases appropriately: - -.Example invocation of complete-run.pl+ +containing the appropriate i3 logfile for each testcase. The latest folder can +always be found under the symlink +latest/+. Unless told differently, it will +run the tests on a separate X server instance (using Xephyr). + +Xephyr will open a window where you can inspect the running test. By default, +tests are run under Xvfb. + +.Example invocation of +complete-run.pl+ --------------------------------------- -$ cd ~/i3/testcases +$ cd ~/i3 + +$ autoreconf -fi -# start two dummy X11 instances in the background -$ ./Xdummy :1 & -$ ./Xdummy :2 & +$ mkdir -p build && cd build -$ ./complete-run.pl -d :1,:2 +$ ../configure + +$ make -j8 +# output omitted because it is very long + +$ cd testcases + +$ ./complete-run.pl # output omitted because it is very long All tests successful. Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU) Result: PASS -$ ./complete-run.pl -d :1 t/04-floating.t +$ ./complete-run.pl t/04-floating.t [:3] i3 startup: took 0.07s, status = 1 [:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t [:3] t/04-floating.t finished @@ -106,6 +166,71 @@ Result: PASS $ less latest/i3-log-for-04-floating.t ---------------------------------------- +If your attempt to run the tests with a bare call to ./complete-run.pl fails, try this: + +--------------------------------------------------- +$ ./complete-run.pl --parallel=1 --keep-xserver-output +--------------------------------------------------- + +This will show the output of Xephyr, which is the X server implementation we +use for testing. + +===== make command: +make check+ +Make check runs the i3 testsuite. +You can still use ./testcases/complete-run.pl to get the interactive progress output. + +.Example invocation of +make check+ +--------------------------------------- +$ cd ~/i3 + +$ autoreconf -fi + +$ mkdir -p build && cd build + +$ ../configure + +$ make -j8 +# output omitted because it is very long + +$ make check +# output omitted because it is very long +PASS: testcases/complete-run.pl +============================================================================ +Testsuite summary for i3 4.13 +============================================================================ +# TOTAL: 1 +# PASS: 1 +# SKIP: 0 +# XFAIL: 0 +# FAIL: 0 +# XPASS: 0 +# ERROR: 0 +============================================================================ + +$ less test-suite.log +---------------------------------------- + +==== Coverage testing + +Coverage testing is possible with +lcov+, the front-end for GCC's coverage +testing tool +gcov+. The testcases can generate a nice html report that tells +you which functions and lines were covered during a run of the tests. You can +use this tool to judge how effective your tests are. + +To use test coverage tools, first compile with coverage enabled. + +--------------------------------------------------- +COVERAGE=1 make +--------------------------------------------------- + +Then run the tests with the +--coverage-testing+ flag. + +--------------------------------------------------- +./complete-run.pl --coverage-testing +--------------------------------------------------- + +Then open +latest/i3-coverage/index.html+ in your web browser. + ==== IPC interface The testsuite makes extensive use of the IPC (Inter-Process Communication) @@ -113,7 +238,7 @@ interface which i3 provides. It is used for the startup process of i3, for terminating it cleanly and (most importantly) for modifying and getting the current state (layout tree). -See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface. +See [https://i3wm.org/docs/ipc.html] for documentation on the IPC interface. ==== X11::XCB @@ -121,23 +246,26 @@ In order to open new windows, change attributes, get events, etc., the testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl module which uses the XCB protocol description to generate Perl bindings to X11. They work in a very similar way to libxcb (which i3 uses) and provide -relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as +relatively high-level interfaces (objects such as +X11::XCB::Window+) as well as access to the low-level interface, which is very useful when testing a window manager. === Filesystem structure In the git root of i3, the testcases live in the folder +testcases+. This -folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base -configuration file which will be used for the tests. The different testcases -(their file extension is .t, not .pl) themselves can be found in the -conventionally named subfolder +t+: +folder contains the +complete-run.pl+ and a base configuration file which will +be used for the tests. The different testcases (their file extension is .t, not +.pl) themselves can be found in the conventionally named subfolder +t+: .Filesystem structure -------------------------------------------- ├── testcases │   ├── complete-run.pl │   ├── i3-test.config +│   ├── lib +│   │   ├── i3test.pm +│   │   ├── SocketActivation.pm +│   │   └── StartXDummy.pm │   ├── t │   │   ├── 00-load.t │   │   ├── 01-tile.t @@ -145,10 +273,7 @@ conventionally named subfolder +t+: │   │   ├── ... │   │   ├── omitted for brevity │   │   ├── ... -│   │   ├── 74-regress-focus-toggle.t -│   │   └── lib -│   │   └── i3test.pm -│   └── Xdummy +│   │   └── 74-regress-focus-toggle.t -------------------------------------------- == Anatomy of a testcase @@ -375,7 +500,7 @@ cmd 'focus left'; is($x->input_focus, $left->id, 'left window focused'); ---------- -However, the test fails. Sometimes. Apparantly, there is a race condition in +However, the test fails. Sometimes. Apparently, there is a race condition in your test. If you think about it, this is because you are using two different pieces of software: You tell i3 to update focus, i3 confirms that, and then you ask X11 to give you the current focus. There is a certain time i3 needs to @@ -449,3 +574,97 @@ request. You should use a random value in +data[1]+ and check that you received the same one when getting the reply. == Appendix B: Socket activation + +Socket activation is a mechanism which was made popular by systemd, an init +replacement. It basically describes creating a listening socket before starting +a program. systemd will invoke the program only when an actual connection to +the socket is made, hence the term socket activation. + +The interesting part of this (in the i3 context) is that you can very precisely +detect when the program is ready (finished its initialization). + +=== Preparing the listening socket + ++complete-run.pl+ will create a listening UNIX socket which it will then pass +to i3. This socket will be used by i3 as an additional IPC socket, just like +the one it will create on its own. Passing the socket happens implicitly +because children will inherit the parent’s sockets when fork()ing and sockets +will continue to exist after an exec() call (unless CLOEXEC is set of course). + +The only explicit things +complete-run.pl+ has to do is setting the +LISTEN_FDS+ +environment variable to the number of sockets which exist (1 in our case) and +setting the +LISTEN_PID+ environment variable to the current process ID. Both +variables are necessary so that the program (i3) knows how many sockets it +should use and if the environment variable is actually intended for it. i3 will +then start looking for sockets at file descriptor 3 (since 0, 1 and 2 are used +for stdin, stdout and stderr, respectively). + +The actual Perl code which sets up the socket, fork()s, makes sure the socket +has file descriptor 3 and sets up the environment variables follows (shortened +a bit): + + +.Setup socket and environment +----------------------------- +my $socket = IO::Socket::UNIX->new( + Listen => 1, + Local => $args{unix_socket_path}, +); + +my $pid = fork; +if ($pid == 0) { + $ENV{LISTEN_PID} = $$; + $ENV{LISTEN_FDS} = 1; + + # Only pass file descriptors 0 (stdin), 1 (stdout), + # 2 (stderr) and 3 (socket) to the child. + $^F = 3; + + # If the socket does not use file descriptor 3 by chance + # already, we close fd 3 and dup2() the socket to 3. + if (fileno($socket) != 3) { + POSIX::close(3); + POSIX::dup2(fileno($socket), 3); + } + + exec "/usr/bin/i3"; +} +----------------------------- + +=== Waiting for a reply + +In the parent process, we want to know when i3 is ready to answer our IPC +requests and handle our windows. Therefore, after forking, we immediately close +the listening socket (i3 will handle this side of the socket) and connect to it +(remember, we are talking about a named UNIX socket) as a client. This connect +call will immediately succeed because the kernel buffers it. Then, we send a +request (of type GET_TREE, but that is not really relevant). Writing data to +the socket will also succeed immediately because, again, the kernel buffers it +(only up to a certain amount of data of course). + +Afterwards, we just blockingly wait until we get an answer. In the child +process, i3 will setup the listening socket in its event loop. Immediately +after actually starting the event loop, it will notice a new client connecting +(the parent process) and handle its request. Since all initialization has been +completed successfully by the time the event loop is entered, we can now assume +that i3 is ready. + +=== Timing and conclusion + +A beautiful feature of this mechanism is that it does not depend on timing. It +does not matter when the child process gets CPU time or when the parent process +gets CPU time. On heavily loaded machines (or machines with multiple CPUs, +cores or unreliable schedulers), this makes waiting for i3 much more robust. + +Before using socket activation, we typically used a +sleep(1)+ and hoped that +i3 was initialized by that time. Of course, this breaks on some (slow) +computers and wastes a lot of time on faster computers. By using socket +activation, we decreased the total amount of time necessary to run all tests +(72 files at the time of writing) from > 100 seconds to 16 seconds. This makes +it significantly more attractive to run the test suite more often (or at all) +during development. + +An alternative approach to using socket activation is polling for the existence +of the IPC socket and connecting to it. While this might be slightly easier to +implement, it wastes CPU time and is considerably uglier than this solution +:). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.