Bacula Regression Suite and CTest
==================================================================
-The Bacula regression scripts have now been modified to use the ctest component
-of cmake. The major gain from this, since Bacula already had a working test
-framework in place, is the ability to have the results of each test submitted
-to a centralized dashboard system. All of the test results are aggregated and
-summarized, where all of the developers can quickly see how the regression
-tests are running.
+Thanks to Frank Sweetser, the Bacula regression scripts have now been modified
+to use the ctest component of cmake. The major gain from this, since Bacula
+already had a working test framework in place, is the ability to have the
+results of each test submitted to a centralized dashboard system. All of the
+test results are aggregated and summarized, where all of the developers can
+quickly see how the regression tests are running.
How to Use CTest
==================================================================
The simplest way to trigger a nightly run is to use one of the two provided
scripts. The nightly-all script will run all non root tests, both tape and
disk based, while the nightly-disk script will run only the disk based tests.
+So, you can choose between the following scripts:
+
+ cd <regress>
+ ./nightly-all # does disk and tape testing
+ ./nightly-disk # disk only tests
+
+ ./experimental-all # experimental disk and tape testing
+ ./experimental-disk # experimental disk testing
+
+If you are a developer and you have modified your local SVN repository, you
+should be running the experimental tests -- they are designed for developers.
+If you do modify your local repository and commit it, then run a nightly
+test, the local repository may be reverted to a prior version so that the
+nightly tests all have a consistent cutoff time.
+
+If you are just doing testing on a nightly basis (no development in your
+source repository), then please use the nightly tests.
+
+All the old scripts (./do_all, do_file, all-non-root-tests, ...) manually
+run the tests outside of ctest.
Periodically, however, you may want to submit a single test separately from a
weekly run. This may be a test of a particular patch you're working on, or
Processing is currently done every 10 minutes, so you may have to wait up to 15
minutes or so before your results show up.
+=========================================================
+
+Email from Frank describing the flow when running a ctest and some of the
+problems that come up.
+
+0. Start off with a local svn repository at version A, and the master
+repository at version B.
+
+1. nightly-disk is started.
+
+2. nightly-disk runs scripts/config_dart
+
+3. config_dart runs scripts/create_sed
+
+4. create_sed pulls bversion and bdate out of the current local repo, so gets
+version A values.
+
+5. config_dart then creates DartConfiguration.tcl from the .in file, leaving a
+BuildName parameter of A.
+
+6. nightly-disk then runs 'ctest -D Nightly'. This implicitly tells ctest to
+perform Update, Configure, Build, Test, and Submit stages, in that order.
+
+7. The Update stage runs 'svn update' on the local repository. The local
+repository is now updated to version B from the master, but since the
+DartConfiguration.tcl file was already created and has not been updated, the
+Update.xml file has the version A BuildName still.
+
+8. Next, the Configure stage runs. Since the configure process is handled in
+tandem with the build process by 'make setup', this just calls /bin/true so as
+to not throw any false errors, and can effectively be treated as a no-op.
+
+9. Next is the Build stage, which is handled by calling scripts/update-ctest.
+
+10. update-ctest checks the svn versions of regress/build vs BACULA_SOURCE.
+Since the two are different (regress/build is still version A, while
+BACULA_SOURCE has been updated to B) it calls 'make setup'.
+
+11. 'make setup' copies BACULA_SOURCE to regress/build and configures and
+builds it. It then calls scripts/do_sed
+
+12. do_sed calls scripts/config_dart again. Since regress/build has been
+updated to B, it regenerates DartConfiguration.tcl with a version B BuildName.
+
+13. ctest now generates the Build.xml results file, but since BuildName was
+still A while this stage began, this is what appears in the XML BuildName.
+
+14. Done with the Build stage, ctest moves on to the Test stage, where it
+actually calls the various test/ scripts as defined by DartTestfile.txt and
+filtered by the -R option. At this point, since ctest is beginning a new
+stage, it appears to re-read DartConfiguration.tcl (I believe this is intended
+to allow ctest to bootstrap itself in a virgin cmake managed source code tree,
+where the test configuration should be generated by cmake). The final
+Test.xml file, therefore, contains a version B BuildName string, as opposed to
+all previous steps, which have version A.
+
+15. Finally, ctest flings the results at the dashboard. The dashboard does
+ignores in what order or grouping XML files are submitted, and instead uses
+the site/buildname tuple to distinguish them (at least for Nightly runs, I'm
+not so sure about Experimental ones). In this case, instead of one complete
+run, the dashboard sees two runs, one of which is Update through Make, and a
+second one which is Test only.
+
+For a sample of the problem, take a look at the fsweetser 2.3 sqlite3 Nightly
+runs at
+
+http://regress.bacula.org:8081/Bacula/Dashboard/Dashboard?trackid=29
+
+The Update and Build information show up with a BuildName of
+bacula-2.3.10-26Feb08-Linux-sqlite3, then after svn update hit the Test
+information shows up with bacula-2.3.11-03Mar08-Linux-sqlite3. (Ignore for
+the moment the fact my timestamps are at 6:59PM, rather than at 9PM where
+they're supposed to be; this seems to be a Fedora specific client side issue I
+haven't tracked down yet.)
+
+Looking more closely at the test submitted to the public dashboard
+(http://public.kitware.com/dashboard.php), I get the impression that the
+BuildName parameter was misnamed, and intended to be treated more as a build
+platform name, rather than the name of the build being tested. Rather than
+create a hook to tag the version being tested, everyone as far as I can tell
+just seems to rely on the timestamp of the test.
+======
+
+NOTE !!!!!!!!! ctest can actually back out changes that have been made to
+your local source repository. As a consequence, it is probably better not to
+use a directory in which you are developing code for Nightly tests. Seee the
+below explanation given by Frank Sweetser.
+
+When a Nightly run is done, the timestamp is set to the last occurring
+instance of the time defined by the NightlyStartTime parameter. The piece
+that I missed is that, in addition to using that timestamp for reporting to
+the dashboard, the update stage also uses that point in time to determine
+exactly which version of the repository to check out.
+
+So if you make commit changes at 10PM EST, and then run a Nightly test run,
+the NightlyStartTime of 9PM EST will back out those changes in the local
+repository. Any subsequent runs that are started at 9PM EST the following day
+or later will include them. This implies to me that NightlyStartTime should
+be set such that you don't expect any developers to commit any changes in
+between NightlyStartTime and the time at which the ctest run actually starts.
+
+The alternative is to make use of the Experimental track. While it normally
+just uses the local source tree as is, you can manually have it update:
+
+ctest -D ExperimentalUpdate
+
+Unlike Nightly, this will update to whatever the latest version of the
+repository is.
projects / bacula / bacula / blobdiff