X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=docs%2Ftestsuite;h=b535e7c141ca994b9f453771097868e9150c8104;hb=b18f507b9b6222f74794022b1fe00491979f589b;hp=e067d33c78694a456e1dd0c491954b6bcf2b4462;hpb=1688ab475ac218878d7807effdf4b44e905161ba;p=i3%2Fi3 diff --git a/docs/testsuite b/docs/testsuite index e067d33c..b535e7c1 100644 --- a/docs/testsuite +++ b/docs/testsuite @@ -1,7 +1,7 @@ i3 testsuite ============ -Michael Stapelberg -October 2011 +Michael Stapelberg +September 2012 This document explains how the i3 testsuite works, how to use it and extend it. It is targeted at developers who not necessarily have been doing testing before @@ -33,6 +33,19 @@ able to easily test if the feature is working correctly. Many developers will test manually if everything works. Having a testcase not only helps you with that, but it will also be useful for every future change. +== Relevant documentation + +Apart from this document, you should also have a look at: + +1. The "Modern Perl" book, which can be found at + http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf +2. The latest Perl documentation of the "i3test" (general testcase setup) and + "i3test::Test" (additional test instructions) modules: + https://build.i3wm.org/docs/lib-i3test.html respectively + https://build.i3wm.org/docs/lib-i3test-test.html +3. The latest documentation on i3’s IPC interface: + https://build.i3wm.org/docs/ipc.html + == Implementation For several reasons, the i3 testsuite has been implemented in Perl: @@ -45,9 +58,49 @@ For several reasons, the i3 testsuite has been implemented in Perl: 2. Perl is widely available and has a well-working package infrastructure. 3. The author is familiar with Perl :). +4. It is a good idea to use a different language for the tests than the + implementation itself. Please do not start programming language flamewars at this point. +=== Installing the dependencies + +As usual with Perl programs, the testsuite ships with a +Makefile.PL+. +This file specifies which Perl modules the testsuite depends on and can be used +to install all of them. + +Perl modules are distributed via CPAN, and there is the official, standard CPAN +client, simply called +cpan+. It comes with every Perl installation and can be +used to install the testsuite. Many users prefer to use the more modern ++cpanminus+ instead, though (because it asks no questions and just works): + +The tests additionally require +Xephyr(1)+ to run a nested X server. Install ++xserver-xephyr+ on Debian or +xorg-server-xephyr+ on Arch Linux. + +.Installing testsuite dependencies using cpanminus (preferred) +-------------------------------------------------------------------------------- +$ cd ~/i3/testcases +$ sudo apt-get install cpanminus +$ sudo cpanm . +$ cd ~/i3/AnyEvent-I3 +$ sudo cpanm Module::Install +$ sudo cpanm . +-------------------------------------------------------------------------------- + +If you don’t want to use cpanminus for some reason, the same works with cpan: + +.Installing testsuite dependencies using cpan +-------------------------------------------------------------------------------- +$ cd ~/i3/testcases +$ sudo cpan . +$ cd ~/i3/AnyEvent-I3 +$ sudo cpan Module::Install +$ sudo cpan . +-------------------------------------------------------------------------------- + +In case you don’t have root permissions, you can also install into your home +directory, see https://michael.stapelberg.de/cpan/ + === Mechanisms ==== Script: complete-run @@ -56,28 +109,35 @@ The testcases are run by a script called +complete-run.pl+. It runs all testcases by default, but you can be more specific and let it only run one or more testcases. Also, it takes care of starting up a separate instance of i3 with an appropriate configuration file and creates a folder for each run -containing the appropriate i3 logfile for each testcase. The latest folder can -always be found under the symlink +latest/+. It is recommended that you run the -tests on one or more separate X server instances (you can only start one window -manager per X session), for example using the provided Xdummy script. -+complete-run.pl+ takes one or more X11 display specifications and parallelizes -the testcases appropriately: - -.Example invocation of complete-run.pl+ +containing the appropriate i3 logfile for each testcase. The latest folder can +always be found under the symlink +latest/+. Unless told differently, it will +run the tests on a separate X server instance (using Xephyr). + +Xephyr will open a window where you can inspect the running test. By default, +tests are run under Xvfb. + +.Example invocation of +complete-run.pl+ --------------------------------------- -$ cd ~/i3/testcases +$ cd ~/i3 + +$ autoreconf -fi + +$ mkdir -p build && cd build + +$ ../configure + +$ make -j8 +# output omitted because it is very long -# start two dummy X11 instances in the background -$ ./Xdummy :1 & -$ ./Xdummy :2 & +$ cd testcases -$ ./complete-run.pl -d :1,:2 +$ ./complete-run.pl # output omitted because it is very long All tests successful. Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU) Result: PASS -$ ./complete-run.pl -d :1 t/04-floating.t +$ ./complete-run.pl t/04-floating.t [:3] i3 startup: took 0.07s, status = 1 [:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t [:3] t/04-floating.t finished @@ -106,6 +166,71 @@ Result: PASS $ less latest/i3-log-for-04-floating.t ---------------------------------------- +If your attempt to run the tests with a bare call to ./complete-run.pl fails, try this: + +--------------------------------------------------- +$ ./complete-run.pl --parallel=1 --keep-xserver-output +--------------------------------------------------- + +This will show the output of Xephyr, which is the X server implementation we +use for testing. + +===== make command: +make check+ +Make check runs the i3 testsuite. +You can still use ./testcases/complete-run.pl to get the interactive progress output. + +.Example invocation of +make check+ +--------------------------------------- +$ cd ~/i3 + +$ autoreconf -fi + +$ mkdir -p build && cd build + +$ ../configure + +$ make -j8 +# output omitted because it is very long + +$ make check +# output omitted because it is very long +PASS: testcases/complete-run.pl +============================================================================ +Testsuite summary for i3 4.13 +============================================================================ +# TOTAL: 1 +# PASS: 1 +# SKIP: 0 +# XFAIL: 0 +# FAIL: 0 +# XPASS: 0 +# ERROR: 0 +============================================================================ + +$ less test-suite.log +---------------------------------------- + +==== Coverage testing + +Coverage testing is possible with +lcov+, the front-end for GCC's coverage +testing tool +gcov+. The testcases can generate a nice html report that tells +you which functions and lines were covered during a run of the tests. You can +use this tool to judge how effective your tests are. + +To use test coverage tools, first compile with coverage enabled. + +--------------------------------------------------- +COVERAGE=1 make +--------------------------------------------------- + +Then run the tests with the +--coverage-testing+ flag. + +--------------------------------------------------- +./complete-run.pl --coverage-testing +--------------------------------------------------- + +Then open +latest/i3-coverage/index.html+ in your web browser. + ==== IPC interface The testsuite makes extensive use of the IPC (Inter-Process Communication) @@ -113,7 +238,7 @@ interface which i3 provides. It is used for the startup process of i3, for terminating it cleanly and (most importantly) for modifying and getting the current state (layout tree). -See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface. +See [https://i3wm.org/docs/ipc.html] for documentation on the IPC interface. ==== X11::XCB @@ -121,23 +246,26 @@ In order to open new windows, change attributes, get events, etc., the testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl module which uses the XCB protocol description to generate Perl bindings to X11. They work in a very similar way to libxcb (which i3 uses) and provide -relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as +relatively high-level interfaces (objects such as +X11::XCB::Window+) as well as access to the low-level interface, which is very useful when testing a window manager. === Filesystem structure In the git root of i3, the testcases live in the folder +testcases+. This -folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base -configuration file which will be used for the tests. The different testcases -(their file extension is .t, not .pl) themselves can be found in the -conventionally named subfolder +t+: +folder contains the +complete-run.pl+ and a base configuration file which will +be used for the tests. The different testcases (their file extension is .t, not +.pl) themselves can be found in the conventionally named subfolder +t+: .Filesystem structure -------------------------------------------- ├── testcases │   ├── complete-run.pl │   ├── i3-test.config +│   ├── lib +│   │   ├── i3test.pm +│   │   ├── SocketActivation.pm +│   │   └── StartXDummy.pm │   ├── t │   │   ├── 00-load.t │   │   ├── 01-tile.t @@ -145,10 +273,7 @@ conventionally named subfolder +t+: │   │   ├── ... │   │   ├── omitted for brevity │   │   ├── ... -│   │   ├── 74-regress-focus-toggle.t -│   │   └── lib -│   │   └── i3test.pm -│   └── Xdummy +│   │   └── 74-regress-focus-toggle.t -------------------------------------------- == Anatomy of a testcase @@ -375,7 +500,7 @@ cmd 'focus left'; is($x->input_focus, $left->id, 'left window focused'); ---------- -However, the test fails. Sometimes. Apparantly, there is a race condition in +However, the test fails. Sometimes. Apparently, there is a race condition in your test. If you think about it, this is because you are using two different pieces of software: You tell i3 to update focus, i3 confirms that, and then you ask X11 to give you the current focus. There is a certain time i3 needs to @@ -539,7 +664,7 @@ activation, we decreased the total amount of time necessary to run all tests it significantly more attractive to run the test suite more often (or at all) during development. -An alternative approach to using socket activation is polling for the existance +An alternative approach to using socket activation is polling for the existence of the IPC socket and connecting to it. While this might be slightly easier to -implement, it wastes CPU time and is considerably more ugly than this solution +implement, it wastes CPU time and is considerably uglier than this solution :). After all, +lib/SocketActivation.pm+ contains only 54 SLOC.