"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">\r
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">\r
<head>\r
-<link rel="icon" type="image/png" href="/favicon.png">\r
+<link rel="icon" type="image/x-icon" href="/favicon.ico">\r
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />\r
-<meta name="generator" content="AsciiDoc 8.6.4" />\r
+<meta name="generator" content="AsciiDoc 8.6.9" />\r
<title>i3: i3 testsuite</title>\r
<link rel="stylesheet" href="/css/style.css" type="text/css" />\r
<link rel="stylesheet" href="/css/xhtml11.css" type="text/css" />\r
<script type="text/javascript">\r
/*<![CDATA[*/\r
-window.onload = function(){asciidoc.footnotes(); asciidoc.toc(2);}\r
+document.addEventListener("DOMContentLoaded", function(){asciidoc.footnotes(); asciidoc.toc(2);}, false);\r
/*]]>*/\r
</script>\r
<script type="text/javascript" src="/js/asciidoc-xhtml11.js"></script>\r
<ul id="nav">\r
<li><a style="border-bottom: 2px solid #fff" href="/docs">Docs</a></li>\r
<li><a href="/screenshots">Screens</a></li>\r
+ <li><a href="https://www.reddit.com/r/i3wm/">FAQ</a></li>\r
<li><a href="/contact">Contact</a></li>\r
- <li><a href="http://bugs.i3wm.org/">Bugs</a></li>\r
+ <li><a href="https://github.com/i3/i3/issues">Bugs</a></li>\r
</ul>\r
<br style="clear: both">\r
<div id="content">\r
<div id="header">\r
<h1>i3 testsuite</h1>\r
<span id="author">Michael Stapelberg</span><br />\r
-<span id="email"><tt><<a href="mailto:michael+i3@stapelberg.de">michael+i3@stapelberg.de</a>></tt></span><br />\r
-<span id="revdate">September 2011</span>\r
+<span id="email"><tt><<a href="mailto:michael@i3wm.org">michael@i3wm.org</a>></tt></span><br />\r
+<span id="revdate">September 2012</span>\r
<div id="toc">
<div id="toctitle">Table of Contents</div>
<noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
</div>\r
</div>\r
<div class="sect1">\r
-<h2 id="_implementation">2. Implementation</h2>\r
+<h2 id="_relevant_documentation">2. Relevant documentation</h2>\r
+<div class="sectionbody">\r
+<div class="paragraph"><p>Apart from this document, you should also have a look at:</p></div>\r
+<div class="olist arabic"><ol class="arabic">\r
+<li>\r
+<p>\r
+The "Modern Perl" book, which can be found at\r
+ <a href="http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf">http://onyxneon.com/books/modern_perl/modern_perl_a4.pdf</a>\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+The latest Perl documentation of the "i3test" (general testcase setup) and\r
+ "i3test::Test" (additional test instructions) modules:\r
+ <a href="http://build.i3wm.org/docs/lib-i3test.html">http://build.i3wm.org/docs/lib-i3test.html</a> respectively\r
+ <a href="http://build.i3wm.org/docs/lib-i3test-test.html">http://build.i3wm.org/docs/lib-i3test-test.html</a>\r
+</p>\r
+</li>\r
+<li>\r
+<p>\r
+The latest documentation on i3’s IPC interface:\r
+ <a href="http://build.i3wm.org/docs/ipc.html">http://build.i3wm.org/docs/ipc.html</a>\r
+</p>\r
+</li>\r
+</ol></div>\r
+</div>\r
+</div>\r
+<div class="sect1">\r
+<h2 id="_implementation">3. Implementation</h2>\r
<div class="sectionbody">\r
<div class="paragraph"><p>For several reasons, the i3 testsuite has been implemented in Perl:</p></div>\r
<div class="olist arabic"><ol class="arabic">\r
The author is familiar with Perl :).\r
</p>\r
</li>\r
+<li>\r
+<p>\r
+It is a good idea to use a different language for the tests than the\r
+ implementation itself.\r
+</p>\r
+</li>\r
</ol></div>\r
<div class="paragraph"><p>Please do not start programming language flamewars at this point.</p></div>\r
<div class="sect2">\r
-<h3 id="_mechanisms">2.1. Mechanisms</h3>\r
+<h3 id="_installing_the_dependencies">3.1. Installing the dependencies</h3>\r
+<div class="paragraph"><p>As usual with Perl programs, the testsuite ships with a <tt>Makefile.PL</tt>.\r
+This file specifies which Perl modules the testsuite depends on and can be used\r
+to install all of them.</p></div>\r
+<div class="paragraph"><p>Perl modules are distributed via CPAN, and there is the official, standard CPAN\r
+client, simply called <tt>cpan</tt>. It comes with every Perl installation and can be\r
+used to install the testsuite. Many users prefer to use the more modern\r
+<tt>cpanminus</tt> instead, though (because it asks no questions and just works):</p></div>\r
+<div class="paragraph"><p>The tests additionally require <tt>Xephyr(1)</tt> to run a nested X server. Install\r
+<tt>xserver-xephyr</tt> on Debian or <tt>xorg-xserver-xephyr</tt> on Arch Linux.</p></div>\r
+<div class="listingblock">\r
+<div class="title">Installing testsuite dependencies using cpanminus (preferred)</div>\r
+<div class="content">\r
+<pre><tt>$ cd ~/i3/testcases\r
+$ sudo apt-get install cpanminus\r
+$ sudo cpanm .\r
+$ cd ~/i3/AnyEvent-I3\r
+$ sudo cpanm .</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>If you don’t want to use cpanminus for some reason, the same works with cpan:</p></div>\r
+<div class="listingblock">\r
+<div class="title">Installing testsuite dependencies using cpan</div>\r
+<div class="content">\r
+<pre><tt>$ cd ~/i3/testcases\r
+$ sudo cpan .\r
+$ cd ~/i3/AnyEvent-I3\r
+$ sudo cpan .</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>In case you don’t have root permissions, you can also install into your home\r
+directory, see <a href="http://michael.stapelberg.de/cpan/">http://michael.stapelberg.de/cpan/</a></p></div>\r
+</div>\r
+<div class="sect2">\r
+<h3 id="_mechanisms">3.2. Mechanisms</h3>\r
<div class="sect3">\r
-<h4 id="_script_complete_run">2.1.1. Script: complete-run</h4>\r
+<h4 id="_script_complete_run">3.2.1. Script: complete-run</h4>\r
<div class="paragraph"><p>The testcases are run by a script called <tt>complete-run.pl</tt>. It runs all\r
testcases by default, but you can be more specific and let it only run one or\r
more testcases. Also, it takes care of starting up a separate instance of i3\r
with an appropriate configuration file and creates a folder for each run\r
containing the appropriate i3 logfile for each testcase. The latest folder can\r
always be found under the symlink <tt>latest/</tt>. Unless told differently, it will\r
-run the tests on a separate X server instance (using the Xdummy script).</p></div>\r
+run the tests on a separate X server instance (using Xephyr).</p></div>\r
+<div class="paragraph"><p>Xephyr will open a window where you can inspect the running test. You can run\r
+the tests without an X session with Xvfb, such as with <tt>xvfb-run\r
+./complete-run</tt>. This will also speed up the tests significantly especially on\r
+machines without a powerful video card.</p></div>\r
<div class="listingblock">\r
-<div class="title">Example invocation of complete-run.pl+</div>\r
+<div class="title">Example invocation of <tt>complete-run.pl</tt></div>\r
<div class="content">\r
-<pre><tt>$ cd ~/i3/testcases\r
+<pre><tt>$ cd ~/i3\r
+\r
+$ autoreconf -fi\r
+\r
+$ mkdir -p build && cd build\r
+\r
+$ ../configure\r
+\r
+$ make -j8\r
+# output omitted because it is very long\r
+\r
+$ cd testcases\r
\r
$ ./complete-run.pl\r
# output omitted because it is very long\r
\r
$ less latest/i3-log-for-04-floating.t</tt></pre>\r
</div></div>\r
+<div class="paragraph"><p>If your attempt to run the tests with a bare call to ./complete-run.pl fails, try this:</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>$ ./complete-run.pl --parallel=1 --keep-xserver-output</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>This will show the output of Xephyr, which is the X server implementation we\r
+use for testing.</p></div>\r
+<div class="sect4">\r
+<h5 id="_make_command_tt_make_check_tt">make command: <tt>make check</tt></h5>\r
+<div class="paragraph"><p>Make check runs the i3 testsuite.\r
+You can still use ./testcases/complete-run.pl to get the interactive progress output.</p></div>\r
+<div class="listingblock">\r
+<div class="title">Example invocation of <tt>make check</tt></div>\r
+<div class="content">\r
+<pre><tt>$ cd ~/i3\r
+\r
+$ autoreconf -fi\r
+\r
+$ mkdir -p build && cd build\r
+\r
+$ ../configure\r
+\r
+$ make -j8\r
+# output omitted because it is very long\r
+\r
+$ make check\r
+# output omitted because it is very long\r
+PASS: testcases/complete-run.pl\r
+============================================================================\r
+Testsuite summary for i3 4.13\r
+============================================================================\r
+# TOTAL: 1\r
+# PASS: 1\r
+# SKIP: 0\r
+# XFAIL: 0\r
+# FAIL: 0\r
+# XPASS: 0\r
+# ERROR: 0\r
+============================================================================\r
+\r
+$ less test-suite.log</tt></pre>\r
+</div></div>\r
+</div>\r
+</div>\r
+<div class="sect3">\r
+<h4 id="_coverage_testing">3.2.2. Coverage testing</h4>\r
+<div class="paragraph"><p>Coverage testing is possible with <tt>lcov</tt>, the front-end for GCC’s coverage\r
+testing tool <tt>gcov</tt>. The testcases can generate a nice html report that tells\r
+you which functions and lines were covered during a run of the tests. You can\r
+use this tool to judge how effective your tests are.</p></div>\r
+<div class="paragraph"><p>To use test coverage tools, first compile with coverage enabled.</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>COVERAGE=1 make</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Then run the tests with the <tt>--coverage-testing</tt> flag.</p></div>\r
+<div class="listingblock">\r
+<div class="content">\r
+<pre><tt>./complete-run.pl --coverage-testing</tt></pre>\r
+</div></div>\r
+<div class="paragraph"><p>Then open <tt>latest/i3-coverage/index.html</tt> in your web browser.</p></div>\r
</div>\r
<div class="sect3">\r
-<h4 id="_ipc_interface">2.1.2. IPC interface</h4>\r
+<h4 id="_ipc_interface">3.2.3. IPC interface</h4>\r
<div class="paragraph"><p>The testsuite makes extensive use of the IPC (Inter-Process Communication)\r
interface which i3 provides. It is used for the startup process of i3, for\r
terminating it cleanly and (most importantly) for modifying and getting the\r
<div class="paragraph"><p>See [<a href="http://i3wm.org/docs/ipc.html">http://i3wm.org/docs/ipc.html</a>] for documentation on the IPC interface.</p></div>\r
</div>\r
<div class="sect3">\r
-<h4 id="_x11_xcb">2.1.3. X11::XCB</h4>\r
+<h4 id="_x11_xcb">3.2.4. X11::XCB</h4>\r
<div class="paragraph"><p>In order to open new windows, change attributes, get events, etc., the\r
testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl\r
module which uses the XCB protocol description to generate Perl bindings to\r
X11. They work in a very similar way to libxcb (which i3 uses) and provide\r
-relatively high-level interfaces (objects such as <tt>X11::XCB::Window</tt>) aswell as\r
+relatively high-level interfaces (objects such as <tt>X11::XCB::Window</tt>) as well as\r
access to the low-level interface, which is very useful when testing a window\r
manager.</p></div>\r
</div>\r
</div>\r
<div class="sect2">\r
-<h3 id="_filesystem_structure">2.2. Filesystem structure</h3>\r
+<h3 id="_filesystem_structure">3.3. Filesystem structure</h3>\r
<div class="paragraph"><p>In the git root of i3, the testcases live in the folder <tt>testcases</tt>. This\r
-folder contains the <tt>complete-run.pl</tt> and <tt>Xdummy</tt> scripts and a base\r
-configuration file which will be used for the tests. The different testcases\r
-(their file extension is .t, not .pl) themselves can be found in the\r
-conventionally named subfolder <tt>t</tt>:</p></div>\r
+folder contains the <tt>complete-run.pl</tt> and a base configuration file which will\r
+be used for the tests. The different testcases (their file extension is .t, not\r
+.pl) themselves can be found in the conventionally named subfolder <tt>t</tt>:</p></div>\r
<div class="listingblock">\r
<div class="title">Filesystem structure</div>\r
<div class="content">\r
│ │ ├── ...\r
│ │ ├── omitted for brevity\r
│ │ ├── ...\r
-│ │ └── 74-regress-focus-toggle.t\r
-│ └── Xdummy</tt></pre>\r
+│ │ └── 74-regress-focus-toggle.t</tt></pre>\r
</div></div>\r
</div>\r
</div>\r
</div>\r
<div class="sect1">\r
-<h2 id="_anatomy_of_a_testcase">3. Anatomy of a testcase</h2>\r
+<h2 id="_anatomy_of_a_testcase">4. Anatomy of a testcase</h2>\r
<div class="sectionbody">\r
<div class="paragraph"><p>Learning by example is definitely a good strategy when you are wondering how to\r
write a testcase. Let’s take <tt>t/11-goto.t</tt> as an easy example and go through it\r
</div>\r
</div>\r
<div class="sect1">\r
-<h2 id="i3_sync">4. Appendix A: The i3 sync protocol</h2>\r
+<h2 id="i3_sync">5. Appendix A: The i3 sync protocol</h2>\r
<div class="sectionbody">\r
<div class="paragraph"><p>Consider the following situation: You open two windows in your testcase, then\r
you use <tt>focus left</tt> and want to verify that the X11 focus has been updated\r
cmd 'focus left';\r
is($x->input_focus, $left->id, 'left window focused');</tt></pre>\r
</div></div>\r
-<div class="paragraph"><p>However, the test fails. Sometimes. Apparantly, there is a race condition in\r
+<div class="paragraph"><p>However, the test fails. Sometimes. Apparently, there is a race condition in\r
your test. If you think about it, this is because you are using two different\r
pieces of software: You tell i3 to update focus, i3 confirms that, and then you\r
ask X11 to give you the current focus. There is a certain time i3 needs to\r
<div class="title">Figure 2. Diagram of the i3 sync solution</div>\r
</div>\r
<div class="sect2">\r
-<h3 id="_implementation_details">4.1. Implementation details</h3>\r
+<h3 id="_implementation_details">5.1. Implementation details</h3>\r
<div class="paragraph"><p>The client which wants to sync with i3 initiates the protocol by sending a\r
ClientMessage to the X11 root window:</p></div>\r
<div class="listingblock">\r
</div>\r
</div>\r
<div class="sect1">\r
-<h2 id="_appendix_b_socket_activation">5. Appendix B: Socket activation</h2>\r
+<h2 id="_appendix_b_socket_activation">6. Appendix B: Socket activation</h2>\r
<div class="sectionbody">\r
<div class="paragraph"><p>Socket activation is a mechanism which was made popular by systemd, an init\r
replacement. It basically describes creating a listening socket before starting\r
<div class="paragraph"><p>The interesting part of this (in the i3 context) is that you can very precisely\r
detect when the program is ready (finished its initialization).</p></div>\r
<div class="sect2">\r
-<h3 id="_preparing_the_listening_socket">5.1. Preparing the listening socket</h3>\r
+<h3 id="_preparing_the_listening_socket">6.1. Preparing the listening socket</h3>\r
<div class="paragraph"><p><tt>complete-run.pl</tt> will create a listening UNIX socket which it will then pass\r
to i3. This socket will be used by i3 as an additional IPC socket, just like\r
the one it will create on its own. Passing the socket happens implicitly\r
</div></div>\r
</div>\r
<div class="sect2">\r
-<h3 id="_waiting_for_a_reply">5.2. Waiting for a reply</h3>\r
+<h3 id="_waiting_for_a_reply">6.2. Waiting for a reply</h3>\r
<div class="paragraph"><p>In the parent process, we want to know when i3 is ready to answer our IPC\r
requests and handle our windows. Therefore, after forking, we immediately close\r
the listening socket (i3 will handle this side of the socket) and connect to it\r
that i3 is ready.</p></div>\r
</div>\r
<div class="sect2">\r
-<h3 id="_timing_and_conclusion">5.3. Timing and conclusion</h3>\r
+<h3 id="_timing_and_conclusion">6.3. Timing and conclusion</h3>\r
<div class="paragraph"><p>A beautiful feature of this mechanism is that it does not depend on timing. It\r
does not matter when the child process gets CPU time or when the parent process\r
gets CPU time. On heavily loaded machines (or machines with multiple CPUs,\r
(72 files at the time of writing) from > 100 seconds to 16 seconds. This makes\r
it significantly more attractive to run the test suite more often (or at all)\r
during development.</p></div>\r
-<div class="paragraph"><p>An alternative approach to using socket activation is polling for the existance\r
+<div class="paragraph"><p>An alternative approach to using socket activation is polling for the existence\r
of the IPC socket and connecting to it. While this might be slightly easier to\r
implement, it wastes CPU time and is considerably uglier than this solution\r
:). After all, <tt>lib/SocketActivation.pm</tt> contains only 54 SLOC.</p></div>\r
projects / i3 / i3.github.io / blobdiff