Bacula Regression Suite and CTest
==================================================================
-The Bacula regression scripts have now been modified to use the ctest component
-of cmake. The major gain from this, since Bacula already had a working test
-framework in place, is the ability to have the results of each test submitted
-to a centralized dashboard system. All of the test results are aggregated and
-summarized, where all of the developers can quickly see how the regression
-tests are running.
+Thanks to Frank Sweetser, the Bacula regression scripts have now been modified
+to use the ctest component of cmake. The major gain from this, since Bacula
+already had a working test framework in place, is the ability to have the
+results of each test submitted to a centralized dashboard system. All of the
+test results are aggregated and summarized, where all of the developers can
+quickly see how the regression tests are running.
How to Use CTest
==================================================================
The simplest way to trigger a nightly run is to use one of the two provided
scripts. The nightly-all script will run all non root tests, both tape and
disk based, while the nightly-disk script will run only the disk based tests.
+So, you can choose between the following scripts:
+
+ cd <regress>
+ ./nightly-all # does disk and tape testing
+ ./nightly-disk # disk only tests
+
+ ./experimental-all # experimental disk and tape testing
+ ./experimental-disk # experimental disk testing
+
+If you are a developer and you have modified your local SVN repository, you
+should be running the experimental tests -- they are designed for developers.
+If you do modify your local repository and commit it, then run a nightly
+test, the local repository may be reverted to a prior version so that the
+nightly tests all have a consistent cutoff time.
+
+If you are just doing testing on a nightly basis (no development in your
+source repository), then please use the nightly tests.
+
+All the old scripts (./do_all, do_file, all-non-root-tests, ...) manually
+run the tests outside of ctest.
Periodically, however, you may want to submit a single test separately from a
weekly run. This may be a test of a particular patch you're working on, or
platform name, rather than the name of the build being tested. Rather than
create a hook to tag the version being tested, everyone as far as I can tell
just seems to rely on the timestamp of the test.
+======
+
+NOTE !!!!!!!!! ctest can actually back out changes that have been made to
+your local source repository. As a consequence, it is probably better not to
+use a directory in which you are developing code for Nightly tests. Seee the
+below explanation given by Frank Sweetser.
+
+When a Nightly run is done, the timestamp is set to the last occurring
+instance of the time defined by the NightlyStartTime parameter. The piece
+that I missed is that, in addition to using that timestamp for reporting to
+the dashboard, the update stage also uses that point in time to determine
+exactly which version of the repository to check out.
+
+So if you make commit changes at 10PM EST, and then run a Nightly test run,
+the NightlyStartTime of 9PM EST will back out those changes in the local
+repository. Any subsequent runs that are started at 9PM EST the following day
+or later will include them. This implies to me that NightlyStartTime should
+be set such that you don't expect any developers to commit any changes in
+between NightlyStartTime and the time at which the ctest run actually starts.
+
+The alternative is to make use of the Experimental track. While it normally
+just uses the local source tree as is, you can manually have it update:
+
+ctest -D ExperimentalUpdate
+
+Unlike Nightly, this will update to whatever the latest version of the
+repository is.