From: Michael Stapelberg Date: Sat, 24 Sep 2011 16:20:36 +0000 (+0100) Subject: Add docs/testsuite X-Git-Tag: 4.1~139^2 X-Git-Url: https://git.sur5r.net/?a=commitdiff_plain;h=d174ff16edc462a7fc44cb1372789bb06ddaa543;p=i3%2Fi3 Add docs/testsuite --- diff --git a/docs/Makefile b/docs/Makefile index 9d70243d..990bba87 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,5 +1,5 @@ -all: hacking-howto.html debugging.html userguide.html ipc.html multi-monitor.html wsbar.html refcard.pdf +all: hacking-howto.html debugging.html userguide.html ipc.html multi-monitor.html wsbar.html refcard.pdf testsuite.html hacking-howto.html: hacking-howto asciidoc -a toc -n $< @@ -10,6 +10,9 @@ debugging.html: debugging userguide.html: userguide asciidoc -a toc -n $< +testsuite.html: testsuite + asciidoc -a toc -n $< + ipc.html: ipc asciidoc -a toc -n $< diff --git a/docs/i3-sync-working.dia b/docs/i3-sync-working.dia new file mode 100644 index 00000000..9f1c3bc8 Binary files /dev/null and b/docs/i3-sync-working.dia differ diff --git a/docs/i3-sync-working.png b/docs/i3-sync-working.png new file mode 100644 index 00000000..dce44ac5 Binary files /dev/null and b/docs/i3-sync-working.png differ diff --git a/docs/i3-sync.dia b/docs/i3-sync.dia new file mode 100644 index 00000000..0945ae26 Binary files /dev/null and b/docs/i3-sync.dia differ diff --git a/docs/i3-sync.png b/docs/i3-sync.png new file mode 100644 index 00000000..b64cce25 Binary files /dev/null and b/docs/i3-sync.png differ diff --git a/docs/testsuite b/docs/testsuite new file mode 100644 index 00000000..7611499d --- /dev/null +++ b/docs/testsuite @@ -0,0 +1,446 @@ +i3 testsuite +============ +Michael Stapelberg +September 2011 + +This document explains how the i3 testsuite works, how to use it and how to +extend it. It is targeted at developers who not necessarily have been doing +testing before or have not been testing in Perl before. In general, the +testsuite is not of interest for end users. + + +== Introduction + +The i3 testsuite is a collection of files which contain testcases for various +i3 features. Some of them test if a certain workflow works correctly (moving +windows, focus behaviour, …). Others are regression tests and contain code +which previously made i3 crash or lead to unexpected behaviour. They then check +if i3 still runs (meaning it did not crash) and if it handled everything +correctly. + +The goal of having these tests is to automatically find problems and to +automatically get a feel for whether a change in the source code breaks any +existing feature. After every modification of the i3 sourcecode, the developer +should run the full testsuite. If one of the tests does not pass (but fails), +the corresponding problem should be fixed (or, in some cases, the testcase has +to be modified). For every bugreport, a testcase should be written to test the +correct behaviour. Initially, it will fail, but after fixing the bug, it will +pass. This ensures (or increases the chance) that bugs which have been fixed +once will never be found again. + +Also, when implementing a new feature, a testcase might be a good way to be +able to easily test if the feature is working correctly. Many developers will +test manually if everything works. Having a testcase not only helps you with +that, but it will also be useful for every future change. + +== Implementation + +For several reasons, the i3 testsuite has been implemented in Perl: + +1. Perl has a long tradition of testing. Every popular/bigger Perl module which + you can find on CPAN will not only come with documentation, but also with + tests. Therefore, the available infrastructure for tests is comprehensive. + See for example the excellent http://search.cpan.org/perldoc?Test::More + and the referenced http://search.cpan.org/perldoc?Test::Tutorial. + +2. Perl is widely available and has a well-working package infrastructure. +3. The author is familiar with Perl :). + +Please do not start programming language flamewars at this point. + +=== Mechanisms + +==== Script: complete-run + +The testcases are run by a script called +complete-run.pl+. It runs all +testcases by default, but you can be more specific and let it only run one or +more testcases. Also, it takes care of starting up a separate instance of i3 +with an appropriate configuration file and creates a folder for each run +containing the appropriate i3 logfile for each testcase. The latest folder can +always be found under the symlink +latest/+. It is recommended that you run the +tests on one or more separate X server instances (you can only start one window +manager per X session), for example using the provided Xdummy script. ++complete-run.pl+ takes one or more X11 display specifications and parallelizes +the testcases appropriately: + +.Example invocation of complete-run.pl+ +--------------------------------------- +$ cd ~/i3/testcases + +# start two dummy X11 instances in the background +$ ./Xdummy :1 & +$ ./Xdummy :2 & + +$ ./complete-run.pl -d :1,:2 +# output omitted because it is very long +All tests successful. +Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU) +Result: PASS + +$ ./complete-run.pl -d :1 t/04-floating.t +[:3] i3 startup: took 0.07s, status = 1 +[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t +[:3] t/04-floating.t finished +[:3] killing i3 +output for t/04-floating.t: +ok 1 - use X11::XCB::Window; +ok 2 - The object isa X11::XCB::Window +ok 3 - Window is mapped +ok 4 - i3 raised the width to 75 +ok 5 - i3 raised the height to 50 +ok 6 - i3 did not map it to (0x0) +ok 7 - The object isa X11::XCB::Window +ok 8 - i3 let the width at 80 +ok 9 - i3 let the height at 90 +ok 10 - i3 mapped it to x=1 +ok 11 - i3 mapped it to y=18 +ok 12 - The object isa X11::XCB::Window +ok 13 - i3 let the width at 80 +ok 14 - i3 let the height at 90 +1..14 + +All tests successful. +Files=1, Tests=14, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.19 cusr 0.03 csys = 0.23 CPU) +Result: PASS + +$ less latest/i3-log-for-04-floating.t +---------------------------------------- + +==== IPC interface + +The testsuite makes extensive use of the IPC (Inter-Process Communication) +interface which i3 provides. It is used for the startup process of i3, for +terminating it cleanly and (most importantly) for modifying and getting the +current state (layout tree). + +See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface. + +==== X11::XCB + +In order to open new windows, change attributes, get events, etc., the +testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl +module which uses the XCB protocol description to generate Perl bindings to +X11. They work in a very similar way to libxcb (which i3 uses) and provide +relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as +access to the low-level interface, which is very useful when testing a window +manager. + +=== Filesystem structure + +In the git root of i3, the testcases live in the folder +testcases+. This +folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base +configuration file which will be used for the tests. The different testcases +themselve can be found in the conventionally named subfolder +t+: + +.Filesystem structure +-------------------------------------------- +├── testcases +│   ├── complete-run.pl +│   ├── i3-test.config +│   ├── t +│   │   ├── 00-load.t +│   │   ├── 01-tile.t +│   │   ├── 02-fullscreen.t +│   │   ├── ... +│   │   ├── omitted for brevity +│   │   ├── ... +│   │   ├── 74-regress-focus-toggle.t +│   │   └── lib +│   │   └── i3test.pm +│   └── Xdummy +-------------------------------------------- + +== Anatomy of a testcase + +Learning by example is definitely a good strategy when you are wondering how to +write a testcase. Let's take +t/11-goto.t+ as an easy example and go through it +step by step: + +.t/11-goto.t: Boilerplate +---------------------- +#!perl +# vim:ts=4:sw=4:expandtab + +use i3test; +use File::Temp; + +my $x = X11::XCB::Connection->new; +----------------------- + +This is what we call boilerplate. It exists at the top of every test file (to +some extent). The first line is the shebang, which specifies that this file is +a Perl script. The second line contains VIM specific settings on how to +edit/format this file (use spaces instead of tabs, indent using 4 spaces). +Afterwards, the +i3test+ module is used. This module contains i3 testsuite +specific functions which you are strongly encouraged to use. They make writing +testcases a lot easier and will make it easier for other people to read your +tests. + +The next line uses the +File::Temp+ module. This is specific to this testcase, +because it needs to generate a temporary name during the test. Many testcases +use only the +i3test+ module. + +The last line opens a connection to X11. You might or might not need this in +your testcase, depending on whether you are going to open windows (etc.) or +only use i3 commands. + +.t/11-goto.t: Setup +---------------------- +my $tmp = fresh_workspace; + +cmd 'split h'; +---------------------- + +The first line calls i3test's +fresh_workspace+ function which looks for a +currently unused workspace, switches to it, and returns its name. The variable ++$tmp+ will end up having a value such as +"/tmp/87kBVcHbA9"+. Note that this +is not (necessarily) a valid path, it's just a random workspace name. + +So, now that we are on a new workspace, we ensure that the workspace uses +horizontal orientation by issuing the +split h+ command (see the i3 User's +Guide for a list of commands). This is not strictly necessary, but good style. +In general, the +cmd+ function executes the specified i3 command by using the +IPC interface and returns once i3 acknowledged the command. + +.t/11-goto.t: Setup +---------------------- +##################################################################### +# Create two windows and make sure focus switching works +##################################################################### + +my $top = open_window($x); +my $mid = open_window($x); +my $bottom = open_window($x); +---------------------- + +In every major section of a testcase, you should put a comment like the one +above. This makes it immediately clear how the file is structured. + +The +open_window+ function opens a standard window, which will then be put into +tiling mode by i3. If you want a floating window, use the ++open_floating_window+ function. These functions accept the same parameters as ++X11::XCB::Window->new+, see the i3test documentation at TODO. + +.t/11-goto.t: Helper function +---------------------- +# +# Returns the input focus after sending the given command to i3 via IPC +# end sleeping for half a second to make sure i3 reacted +# +sub focus_after { + my $msg = shift; + + cmd $msg; + sync_with_i3 $x; + return $x->input_focus; +} +---------------------- + +This section defines a helper function which will be used over and over in this +testcase. If you have code which gets executed more than once or twice +(depending on the length of your test, use your best judgement), please put it +in a function. Tests should be short, concise and clear. + +The +focus_after+ function executes a command and returns the X11 focus after +the command was executed. The +sync_with_i3+ command makes sure that i3 could +push its state to X11. See <> to learn how this works exactly. + +.t/11-goto.t: Test assumptions +---------------------- +$focus = $x->input_focus; +is($focus, $bottom->id, "Latest window focused"); + +$focus = focus_after('focus left'); +is($focus, $mid->id, "Middle window focused"); +---------------------- + +Now, we run the first two real tests. They use +Test::More+'s +is+ function, +which compares two values and prints the differences if they are not the same. +After the arguments, we supply a short comment to indicate what we are testing +here. This makes it vastly more easy for the developer to spot which testcase +is the problem in case one fails. + +The first test checks that the most recently opened window is focused. +Afterwards, the command +focus left+ is issued and it is verified that the +middle window now has focus. + +Note that this is not a comprehensive test of the +focus+ command -- we would +have to test wrapping, focus when using a more complex layout, focusing the +parent/child containers, etc. But that is not the point of this testcase. +Instead, we just want to know if +$x->input_focus+ corresponds with what we are +expecting. If not, something is completely wrong with the test environment and +this trivial test will fail. + +.t/11-goto.t: Test that the feature does not work (yet) +---------------------- +##################################################################### +# Now goto a mark which does not exist +##################################################################### + +my $random_mark = mktemp('mark.XXXXXX'); + +$focus = focus_after(qq|[con_mark="$random_mark"] focus|); +is($focus, $mid->id, "focus unchanged"); +---------------------- + +In this new major section, a random mark (mark is an identifier for a window, +see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we +test that trying to focus that mark will not do anything. This is important: Do +not only test that using a feature has the expected outcome, but also test that +using it without properly initializing it does no harm. This command could for +example have changed focus anyways (a bug) or crash i3 (obviously a bug). + +.t/11-goto.t: Test that the feature does work +---------------------- +cmd "mark $random_mark"; + +$focus = focus_after('focus left'); +is($focus, $top->id, "Top window focused"); + +$focus = focus_after(qq|[con_mark="$random_mark"] focus|); +is($focus, $mid->id, "goto worked"); +---------------------- + +Remember: Focus was on the middle window (we verified that earlier in "Test +assumptions"). We now mark the middle window with our randomly generated mark. +Afterwards, we switch focus away from the middle window to be able to tell if +focusing it via its mark will work. If it does work (next test), the goto +command works. + +.t/11-goto.t: Test corner case +---------------------- +# check that we can specify multiple criteria + +$focus = focus_after('focus left'); +is($focus, $top->id, "Top window focused"); + +$focus = focus_after(qq|[con_mark="$random_mark" con_mark="$random_mark"] focus|); +is($focus, $mid->id, "goto worked"); +---------------------- + +Now we test the same feature, but specifying the mark twice in the command. +This should have no effect, but let’s be sure: test it and see if things go +wrong. + +.t/11-goto.t: Test second code path +---------------------- +##################################################################### +# Check whether the focus command will switch to a different +# workspace if necessary +##################################################################### + +my $tmp2 = fresh_workspace; + +is(focused_ws(), $tmp2, 'tmp2 now focused'); + +cmd qq|[con_mark="$random_mark"] focus|; + +is(focused_ws(), $tmp, 'tmp now focused'); +---------------------- + +This part of the test checks that focusing windows by mark works across +workspaces. It uses i3test's +focused_ws+ function to get the current +workspace. + +.t/11-goto.t: Test second code path +---------------------- +done_testing; +---------------------- + +The end of every testcase has to contain the +done_testing+ line. This tells ++complete-run.pl+ that the test was finished successfully. If it does not +occur, the test might have crashed during execution -- some of the reasons why +that could happen are bugs in the used modules, bugs in the testcase itself or +an i3 crash resulting in the testcase being unable to communicate with i3 via +IPC anymore. + +[[i3_sync]] +== Appendix A: The i3 sync protocol + +Consider the following situation: You open two windows in your testcase, then +you use +focus left+ and want to verify that the X11 focus has been updated +properly. Sounds simple, right? Let’s assume you use this straight-forward +implementation: + +.Racey focus testcase +----------- +my $left = open_window($x); +my $right = open_window($x); +cmd 'focus left'; +is($x->input_focus, $left->id, 'left window focused'); +---------- + +However, the test fails. Sometimes. Apparantly, there is a race condition in +your test. If you think about it, this is because you are using two different +pieces of software: You tell i3 to update focus, i3 confirms that, and then you +ask X11 to give you the current focus. There is a certain time i3 needs to +update the X11 state. If the testcase gets CPU time before X11 processed i3's +requests, the test will fail. + +image::i3-sync.png["Diagram of the race condition", title="Diagram of the race condition"] + +One way to "solve" this would be to add +sleep 0.5;+ after the +cmd+ call. +After 0.5 seconds it should be safe to assume that focus has been updated, +right? + +In practice, this usually works. However, it has several problems: + +1. This is obviously not a clean solution, but a workaround. Ugly. +2. On very slow machines, this might not work. Unlikely, but in different + situations (a delay to wait for i3 to startup) the necessary time is much + harder to guess, even for fast machines. +3. This *wastes a lot of time*. Usually, your computer is much faster than 0.5s + to update the status. However, sometimes, it might take 0.4s, so we can’t + make it +sleep 0.1+. + +To illustrate how grave the problem with wasting time actually is: Before +removing all sleeps from the testsuite, a typical run using 4 separate X +servers took around 50 seconds on my machine. After removing all the sleeps, +we achieved times of about 25 seconds. This is very significant and influences +the way you think about tests -- the faster they are, the more likely you are +to check whether everything still works quite often (which you should). + +What I am trying to say is: This adds up quickly and makes the test suite less +robust. + +The real solution for this problem is a mechanism which I call "the i3 sync +protocol". The idea is to send a request (which does not modify state) via X11 +to i3 which will then be answered. Due to the request's position in the event +queue (*after* all previous events), you can be sure that by the time you +receive the reply, all other events have been dealt with by i3 (and, more +importantly, X11). + +image::i3-sync-working.png["Diagram of the i3 sync solution", title="Diagram of the i3 sync solution"] + +=== Implementation details + +The client which wants to sync with i3 initiates the protocol by sending a +ClientMessage to the X11 root window: + +.Send ClientMessage +------------------- +# Generate a ClientMessage, see xcb_client_message_t +my $msg = pack "CCSLLLLLLL", + CLIENT_MESSAGE, # response_type + 32, # format + 0, # sequence + $root, # destination window + $x->atom(name => 'I3_SYNC')->id, + + $_sync_window->id, # data[0]: our own window id + $myrnd, # data[1]: a random value to identify the request + 0, + 0, + 0; + +# Send it to the root window -- since i3 uses the SubstructureRedirect +# event mask, it will get the ClientMessage. +$x->send_event(0, $root, EVENT_MASK_SUBSTRUCTURE_REDIRECT, $msg); +------------------- + +i3 will then reply with the same ClientMessage, sent to the window specified in ++data[0]+. In the reply, +data[0]+ and +data[1]+ are exactly the same as in the +request. You should use a random value in +data[1]+ and check that you received +the same one when getting the reply. + +== Appendix B: Socket activation