Bacula Regression Suite and CTest
==================================================================

The Bacula regression scripts have now been modified to use the ctest component
of cmake.  The major gain from this, since Bacula already had a working test
framework in place, is the ability to have the results of each test submitted
to a centralized dashboard system.  All of the test results are aggregated and
summarized, where all of the developers can quickly see how the regression
tests are running.

How to Use CTest
==================================================================

For more complete documentation on ctest, go to:

http://www.cmake.org/Wiki/CMake#CTest

The first step is to install the cmake package, which includes ctest.  If your
distribution does already package it, you can download it directly from the
source:

http://www.cmake.org

Next, you must edit your regression config file and add a paramter called
SITE_NAME to identify the machine running the tests.  Ideally, it should
contain something to identify yourself to whoever is viewing the test results
as well as something to allow you to identify which machine is running the
tests, ie

SITE_NAME=kern-bacula-gumbie

Once you have cmake installed, you can perform one of two different kinds of
runs to submit test results.  The most common kind will be Nightly runs.  A
Nightly CTest backup will update the source directory (as defined by the
BACULA_SOURCE setting in your config file) to the current version, run the
specified list of tests, and submit all of the results to the server.  Note
that all of the results in a given 24 hour period (starting at 9pm EST) are
lumped together to appear as a single block, rather than each test showing up
as a different run.

The simplest way to trigger a nightly run is to use one of the two provided
scripts.  The nightly-all script will run all non root tests, both tape and
disk based, while the nightly-disk script will run only the disk based tests.

Periodically, however, you may want to submit a single test separately from a
weekly run.  This may be a test of a particular patch you're working on, or
perhaps a new OS patch.  For these one-shot tests, you will want to manually
run ctest in Experimental mode, something like:

REGRESS_DEBUG=1 ctest -D Experimental -R all-non-root:auto-label-test

The '-D Experimental' option tells ctest to submit the test results as
Experimental instead of Nightly.  We reccomend you use the REGRESS_DEBUG
environmental variable to ensure that any errors from the test are logged in
the dashboard (all of the ctest wrapper scripts set it).  The '-R <pattern>'
option gives ctest a regular expression.  Any tests with a name as defined in
DartTestfile.txt that matches the pattern will be run.

Note that you must have run ./scripts/do_sed at least once already in order to
use Experimental mode.

Updating and Building Within CTest
==================================================================

Before each Nightly run, ctest will automatically update the BACULA_SOURCE
directory, and submit these updates along with the test results.  Any
Experimental runs will not.

Before either type of run actually begins running tests, ctest will run the
script scripts/update-ctest.  This script first compares the svn version of
BUILD_SOURCE with that of the build/ directory.  If the two versions differ, or
if the build/ directory does not exist, it will automatically run 'make setup'
for you.

Viewing the Dashboard
==================================================================

You can view the dashboard at:

http://regress.bacula.org:8081

Results will not be visible as soon as they are submitted to the server.
Processing is currently done every 10 minutes, so you may have to wait up to 15
minutes or so before your results show up.