2 Bacula Regression Suite and CTest
3 ==================================================================
5 The Bacula regression scripts have now been modified to use the ctest component
6 of cmake. The major gain from this, since Bacula already had a working test
7 framework in place, is the ability to have the results of each test submitted
8 to a centralized dashboard system. All of the test results are aggregated and
9 summarized, where all of the developers can quickly see how the regression
13 ==================================================================
15 For more complete documentation on ctest, go to:
17 http://www.cmake.org/Wiki/CMake#CTest
19 The first step is to install the cmake package, which includes ctest. If your
20 distribution does already package it, you can download it directly from the
25 Next, you must edit your regression config file and add a paramter called
26 SITE_NAME to identify the machine running the tests. Ideally, it should
27 contain something to identify yourself to whoever is viewing the test results
28 as well as something to allow you to identify which machine is running the
31 SITE_NAME=kern-bacula-gumbie
33 Once you have cmake installed, you can perform one of two different kinds of
34 runs to submit test results. The most common kind will be Nightly runs. A
35 Nightly CTest backup will update the source directory (as defined by the
36 BACULA_SOURCE setting in your config file) to the current version, run the
37 specified list of tests, and submit all of the results to the server. Note
38 that all of the results in a given 24 hour period (starting at 9pm EST) are
39 lumped together to appear as a single block, rather than each test showing up
42 The simplest way to trigger a nightly run is to use one of the two provided
43 scripts. The nightly-all script will run all non root tests, both tape and
44 disk based, while the nightly-disk script will run only the disk based tests.
46 Periodically, however, you may want to submit a single test separately from a
47 weekly run. This may be a test of a particular patch you're working on, or
48 perhaps a new OS patch. For these one-shot tests, you will want to manually
49 run ctest in Experimental mode, something like:
51 REGRESS_DEBUG=1 ctest -D Experimental -R all-non-root:auto-label-test
53 The '-D Experimental' option tells ctest to submit the test results as
54 Experimental instead of Nightly. We reccomend you use the REGRESS_DEBUG
55 environmental variable to ensure that any errors from the test are logged in
56 the dashboard (all of the ctest wrapper scripts set it). The '-R <pattern>'
57 option gives ctest a regular expression. Any tests with a name as defined in
58 DartTestfile.txt that matches the pattern will be run.
60 Note that you must have run ./scripts/do_sed at least once already in order to
61 use Experimental mode.
63 Updating and Building Within CTest
64 ==================================================================
66 Before each Nightly run, ctest will automatically update the BACULA_SOURCE
67 directory, and submit these updates along with the test results. Any
68 Experimental runs will not.
70 Before either type of run actually begins running tests, ctest will run the
71 script scripts/update-ctest. This script first compares the svn version of
72 BUILD_SOURCE with that of the build/ directory. If the two versions differ, or
73 if the build/ directory does not exist, it will automatically run 'make setup'
77 ==================================================================
79 You can view the dashboard at:
81 http://regress.bacula.org:8081
83 Results will not be visible as soon as they are submitted to the server.
84 Processing is currently done every 10 minutes, so you may have to wait up to 15
85 minutes or so before your results show up.
87 =========================================================
89 Email from Frank describing the flow when running a ctest and some of the
90 problems that come up.
92 0. Start off with a local svn repository at version A, and the master
93 repository at version B.
95 1. nightly-disk is started.
97 2. nightly-disk runs scripts/config_dart
99 3. config_dart runs scripts/create_sed
101 4. create_sed pulls bversion and bdate out of the current local repo, so gets
104 5. config_dart then creates DartConfiguration.tcl from the .in file, leaving a
105 BuildName parameter of A.
107 6. nightly-disk then runs 'ctest -D Nightly'. This implicitly tells ctest to
108 perform Update, Configure, Build, Test, and Submit stages, in that order.
110 7. The Update stage runs 'svn update' on the local repository. The local
111 repository is now updated to version B from the master, but since the
112 DartConfiguration.tcl file was already created and has not been updated, the
113 Update.xml file has the version A BuildName still.
115 8. Next, the Configure stage runs. Since the configure process is handled in
116 tandem with the build process by 'make setup', this just calls /bin/true so as
117 to not throw any false errors, and can effectively be treated as a no-op.
119 9. Next is the Build stage, which is handled by calling scripts/update-ctest.
121 10. update-ctest checks the svn versions of regress/build vs BACULA_SOURCE.
122 Since the two are different (regress/build is still version A, while
123 BACULA_SOURCE has been updated to B) it calls 'make setup'.
125 11. 'make setup' copies BACULA_SOURCE to regress/build and configures and
126 builds it. It then calls scripts/do_sed
128 12. do_sed calls scripts/config_dart again. Since regress/build has been
129 updated to B, it regenerates DartConfiguration.tcl with a version B BuildName.
131 13. ctest now generates the Build.xml results file, but since BuildName was
132 still A while this stage began, this is what appears in the XML BuildName.
134 14. Done with the Build stage, ctest moves on to the Test stage, where it
135 actually calls the various test/ scripts as defined by DartTestfile.txt and
136 filtered by the -R option. At this point, since ctest is beginning a new
137 stage, it appears to re-read DartConfiguration.tcl (I believe this is intended
138 to allow ctest to bootstrap itself in a virgin cmake managed source code tree,
139 where the test configuration should be generated by cmake). The final
140 Test.xml file, therefore, contains a version B BuildName string, as opposed to
141 all previous steps, which have version A.
143 15. Finally, ctest flings the results at the dashboard. The dashboard does
144 ignores in what order or grouping XML files are submitted, and instead uses
145 the site/buildname tuple to distinguish them (at least for Nightly runs, I'm
146 not so sure about Experimental ones). In this case, instead of one complete
147 run, the dashboard sees two runs, one of which is Update through Make, and a
148 second one which is Test only.
150 For a sample of the problem, take a look at the fsweetser 2.3 sqlite3 Nightly
153 http://regress.bacula.org:8081/Bacula/Dashboard/Dashboard?trackid=29
155 The Update and Build information show up with a BuildName of
156 bacula-2.3.10-26Feb08-Linux-sqlite3, then after svn update hit the Test
157 information shows up with bacula-2.3.11-03Mar08-Linux-sqlite3. (Ignore for
158 the moment the fact my timestamps are at 6:59PM, rather than at 9PM where
159 they're supposed to be; this seems to be a Fedora specific client side issue I
160 haven't tracked down yet.)
162 Looking more closely at the test submitted to the public dashboard
163 (http://public.kitware.com/dashboard.php), I get the impression that the
164 BuildName parameter was misnamed, and intended to be treated more as a build
165 platform name, rather than the name of the build being tested. Rather than
166 create a hook to tag the version being tested, everyone as far as I can tell
167 just seems to rely on the timestamp of the test.