Tweaks from Dan

[bacula/docs] / docs / manuals / en / developers / regression.tex

%%
%%

\chapter{Bacula Regression Testing}
\label{_ChapterStart8}
\index{Testing!Bacula Regression}
\index{Bacula Regression Testing}
\addcontentsline{toc}{section}{Bacula Regression Testing}

\section{Setting up Regession Testing}
\index{Setting up Regession Testing}
\addcontentsline{toc}{section}{Setting up Regression Testing}

This document is intended mostly for developers who wish to ensure that their
changes to Bacula don't introduce bugs in the base code.  However, you
don't need to be a developer to run the regression scripts, and we 
recommend them before putting your system into production, and before each
upgrade, especially if you build from source code.  They are
simply shell scripts that drive Bacula through bconsole and then typically
compare the input and output with {\bf diff}.

You can find the existing regression scripts in the Bacula developer's
{\bf git} repository on SourceForge.  We strongly recommend that you {\bf
clone} the repository because afterwards, you can easily get pull the
updates that have been made.

To get started, we recommend that you create a directory named {\bf
bacula}, under which you will put the current source code and the current
set of regression scripts.  Below, we will describe how to set this up.

The top level directory that we call {\bf bacula} can be named anything you
want.  Note, all the standard regression scripts run as non-root and can be
run on the same machine as a production Bacula system (the developers run
it this way).

To create the directory structure for the current trunk and to
clone the repository, do the following (note, we assume you
are working in your home directory in a non-root account):

\footnotesize
\begin{verbatim}
git clone http://git.bacula.org/bacula bacula
\end{verbatim}
\normalsize

This will create the directory {\bf bacula} and populate it with
three directories: {\bf bacula}, {\bf gui}, and {\bf regress}.
{\bf bacula} contains the Bacula source code; {\bf gui} contains
certain gui programs that you will not need, and {\bf regress} contains
all the regression scripts.  The above should be needed only
once. Thereafter to update to the latest code, you do:

\footnotesize
\begin{verbatim}
cd bacula
git pull
\end{verbatim}
\normalsize

If you want to test with SQLite and it is not installed on your system,
you will need to download the latest depkgs release from Source Forge and       
unpack it into {\bf depkgs}, then simply: 

\footnotesize
\begin{verbatim}
cd depkgs
make
\end{verbatim}
\normalsize


There are two different aspects of regression testing that this document will
discuss: 1. Running the Regression Script, 2. Writing a Regression test. 

\section{Running the Regression Script}
\index{Running the Regression Script}
\index{Script!Running the Regression}
\addcontentsline{toc}{section}{Running the Regression Script}

There are a number of different tests that may be run, such as: the standard
set that uses disk Volumes and runs under any userid; a small set of tests
that write to tape; another set of tests where you must be root to run them.
Normally, I run all my tests as non-root and very rarely run the root
tests.  The tests vary in length, and running the full tests including disk
based testing, tape based testing, autochanger based testing, and multiple
drive autochanger based testing can take 3 or 4 hours.

\subsection{Setting the Configuration Parameters}
\index{Setting the Configuration Parameters}
\index{Parameters!Setting the Configuration}
\addcontentsline{toc}{subsection}{Setting the Configuration Parameters}

There is nothing you need to change in the source directory.  
 
To begin:

\footnotesize
\begin{verbatim}
cd bacula/regress
\end{verbatim}
\normalsize


The        
very first time you are going to run the regression scripts, you will
need to create a custom config file for your system. 
We suggest that you start by:

\footnotesize
\begin{verbatim}
cp prototype.conf config
\end{verbatim}
\normalsize

Then you can edit the {\bf config} file directly.

\footnotesize
\begin{verbatim}
                                                                                        
# Where to get the source to be tested
BACULA_SOURCE="${HOME}/bacula/bacula"

# Where to send email   !!!!! Change me !!!!!!!
EMAIL=your-name@your-domain.com
SMTP_HOST="localhost"

# Full "default" path where to find sqlite (no quotes!)
SQLITE3_DIR=${HOME}/depkgs/sqlite3
SQLITE_DIR=${HOME}/depkgs/sqlite

TAPE_DRIVE="/dev/nst0"
# if you don't have an autochanger set AUTOCHANGER to /dev/null
AUTOCHANGER="/dev/sg0"
# For two drive tests -- set to /dev/null if you do not have it 
TAPE_DRIVE1="/dev/null"

# This must be the path to the autochanger including its name
AUTOCHANGER_PATH="/usr/sbin/mtx"

# Set what backend to use "postresql" "mysql" or "sqlite3"
DBTYPE="postgresql"

# Set your database here
#WHICHDB="--with-${DBTYPE}=${SQLITE3_DIR}"
WHICHDB="--with-${DBTYPE}"

# Set this to "--with-tcp-wrappers" or "--without-tcp-wrappers"
TCPWRAPPERS="--with-tcp-wrappers"

# Set this to "" to disable OpenSSL support, "--with-openssl=yes"
# to enable it, or provide the path to the OpenSSL installation,
# eg "--with-openssl=/usr/local"
OPENSSL="--with-openssl"

# You may put your real host name here, but localhost or 127.0.0.1
# is valid also and it has the advantage that it works on a 
# non-networked machine
HOST="localhost"
                                                                                        
\end{verbatim}
\normalsize

\begin{itemize}
\item {\bf BACULA\_SOURCE} should be the full path to the Bacula source code 
   that you wish to test. It will be loaded configured, compiled, and
   installed with the "make setup" command, which needs to be done only
   once each time you change the source code.

\item {\bf EMAIL} should be your email addres. Please remember  to change this
   or I will get a flood of unwanted  messages. You may or may not want to see
   these emails. In  my case, I don't need them so I direct it to the bit bucket.

\item {\bf SMTP\_HOST} defines where your SMTP server is.

\item {\bf SQLITE\_DIR} should be the full path to the sqlite package,  must
   be build before running a Bacula regression, if you are  using SQLite. This
   variable is ignored if you are using  MySQL or PostgreSQL. To use PostgreSQL,
   edit the Makefile  and change (or add) WHICHDB?=``\verb{--{with-postgresql''.  For
   MySQL use ``WHICHDB=''\verb{--{with-mysql``. 
   
   The advantage of using SQLite is that it is totally independent of any
   installation you may have running on your system, and there is no
   special configuration or authorization that must be done to run it.
   With both MySQL and PostgreSQL, you must pre-install the packages,
   initialize them and ensure that you have authorization to access the 
   database and create and delete tables.

\item {\bf TAPE\_DRIVE} is the full path to your tape drive.  The base set of
   regression tests do not use a tape, so this is only important if you want to
   run the full tests. Set this to  /dev/null if you do not have a tape drive.

\item {\bf TAPE\_DRIVE1} is the full path to your second tape drive, if
   have one. The base set of
   regression tests do not use a tape, so  this is only important if you want to
   run the full two drive tests.  Set this to  /dev/null if you do not have a
   second tape drive.

\item {\bf AUTOCHANGER} is the name of your autochanger control device.  Set this to
   /dev/null if you do not have an autochanger. 

\item {\bf AUTOCHANGER\_PATH} is the full path including the  program name for
   your autochanger program (normally  {\bf mtx}. Leave the default value if you
   do not have one. 

\item {\bf TCPWRAPPERS} defines whether or not you want the ./configure
   to be performed with tcpwrappers enabled.

\item {\bf OPENSSL} used to enable/disable SSL support for Bacula
   communications and data encryption.

\item {\bf HOST} is the hostname that it will use when building the
   scripts. The Bacula daemons will be named <HOST>-dir, <HOST>-fd,
   ... It is also the name of the HOST machine that to connect to the
   daemons by the network.  Hence the name should either be your real
   hostname (with an appropriate DNS or /etc/hosts entry) or {\bf
   localhost} as it is in the default file.

\item {\bf bin} is the binary location.

\item {\bf scripts} is the bacula scripts location (where we could find
  database creation script, autochanger handler, etc.)

\end{itemize}

\subsection{Building the Test Bacula}
\index{Building the Test Bacula}
\index{Bacula!Building the Test}
\addcontentsline{toc}{subsection}{Building the Test Bacula}

Once the above variables are set, you can build the setup by entering:

\footnotesize
\begin{verbatim}
make setup
\end{verbatim}
\normalsize

This will setup the regression testing and you should not need to
do this again unless you want to change the database or other regression
configuration parameters.


\subsection{Setting up your SQL engine}
\index{Setting up your SQL engine}
\addcontentsline{toc}{subsection}{Setting up your SQL engine}
If you are using SQLite or SQLite3, there is nothing more to do; you can   
simply run the tests as described in the next section.

If you are using MySQL or PostgreSQL, you will need to establish an
account with your database engine for the user name {\bf regress} and
you will need to manually create a database named {\bf regress} that can be
used by user name regress, which means you will have to give the user
regress sufficient permissions to use the database named regress.
There is no password on the regress account.

You have probably already done this procedure for the user name and
database named bacula.  If not, the manual describes roughly how to
do it, and the scripts in bacula/regress/build/src/cats named
create\_mysql\_database, create\_postgresql\_database, grant\_mysql\_privileges,
and grant\_postgresql\_privileges may be of a help to you.

Generally, to do the above, you will need to run under root to 
be able to create databases and modify permissions within MySQL and
PostgreSQL.

It is possible to configure MySQL access for database accounts that
require a password to be supplied. This can be done by creating a ~/.my.cnf
file which supplies the credentials by default to the MySQL commandline
utilities.

\begin{verbatim}
[client]
host     = localhost
user     = regress
password = asecret
\end{verbatim}

A similar technique can be used PostgreSQL regression testing where the
database is configured to require a password. The ~/.pgpass file should
contain a line with the database connection properties.

\begin{verbatim}
hostname:port:database:username:password
\end{verbatim}

\subsection{Running the Disk Only Regression}
\index{Regression!Running the Disk Only}
\index{Running the Disk Only Regression}
\addcontentsline{toc}{subsection}{Running the Disk Only Regression}

The simplest way to copy the source code, configure it, compile it, link
it, and run the tests is to use a helper script:

\footnotesize
\begin{verbatim}
./do_disk
\end{verbatim}
\normalsize




This will run the base set of tests using disk Volumes.
If you are testing on a
non-Linux machine several of the of the tests may not be run.  In any case,
as we add new tests, the number will vary.  It will take about 1 hour
and you don't need to be root
to run these tests (I run under my regular userid).  The result should be
something similar to:

\footnotesize
\begin{verbatim}
Test results
  ===== auto-label-test OK 12:31:33 =====
  ===== backup-bacula-test OK 12:32:32 =====
  ===== bextract-test OK 12:33:27 =====
  ===== bscan-test OK 12:34:47 =====
  ===== bsr-opt-test OK 12:35:46 =====
  ===== compressed-test OK 12:36:52 =====
  ===== compressed-encrypt-test OK 12:38:18 =====
  ===== concurrent-jobs-test OK 12:39:49 =====
  ===== data-encrypt-test OK 12:41:11 =====
  ===== encrypt-bug-test OK 12:42:00 =====
  ===== fifo-test OK 12:43:46 =====
  ===== backup-bacula-fifo OK 12:44:54 =====
  ===== differential-test OK 12:45:36 =====
  ===== four-concurrent-jobs-test OK 12:47:39 =====
  ===== four-jobs-test OK 12:49:22 =====
  ===== incremental-test OK 12:50:38 =====
  ===== query-test OK 12:51:37 =====
  ===== recycle-test OK 12:53:52 =====
  ===== restore2-by-file-test OK 12:54:53 =====
  ===== restore-by-file-test OK 12:55:40 =====
  ===== restore-disk-seek-test OK 12:56:29 =====
  ===== six-vol-test OK 12:57:44 =====
  ===== span-vol-test OK 12:58:52 =====
  ===== sparse-compressed-test OK 13:00:00 =====
  ===== sparse-test OK 13:01:04 =====
  ===== two-jobs-test OK 13:02:39 =====
  ===== two-vol-test OK 13:03:49 =====
  ===== verify-vol-test OK 13:04:56 =====
  ===== weird-files2-test OK 13:05:47 =====
  ===== weird-files-test OK 13:06:33 =====
  ===== migration-job-test OK 13:08:15 =====
  ===== migration-jobspan-test OK 13:09:33 =====
  ===== migration-volume-test OK 13:10:48 =====
  ===== migration-time-test OK 13:12:59 =====
  ===== hardlink-test OK 13:13:50 =====
  ===== two-pool-test OK 13:18:17 =====
  ===== fast-two-pool-test OK 13:24:02 =====
  ===== two-volume-test OK 13:25:06 =====
  ===== incremental-2disk OK 13:25:57 =====
  ===== 2drive-incremental-2disk OK 13:26:53 =====
  ===== scratch-pool-test OK 13:28:01 =====
Total time = 0:57:55 or 3475 secs

\end{verbatim}
\normalsize

and the working tape tests are run with

\footnotesize
\begin{verbatim}
make full_test
\end{verbatim}
\normalsize


\footnotesize
\begin{verbatim}
Test results
  
  ===== Bacula tape test OK =====
  ===== Small File Size test OK =====
  ===== restore-by-file-tape test OK =====
  ===== incremental-tape test OK =====
  ===== four-concurrent-jobs-tape OK =====
  ===== four-jobs-tape OK =====
\end{verbatim}
\normalsize

Each separate test is self contained in that it initializes to run Bacula from
scratch (i.e. newly created database). It will also kill any Bacula session
that is currently running. In addition, it uses ports 8101, 8102, and 8103 so
that it does not intefere with a production system. 

Alternatively, you can do the ./do\_disk work by hand with:

\footnotesize
\begin{verbatim}
make setup
\end{verbatim}
\normalsize

The above will then copy the source code within
the regression tree (in directory regress/build), configure it, and build it.
There should be no errors. If there are, please correct them before
continuing. From this point on, as long as you don't change the Bacula
source code, you should not need to repeat any of the above steps.  If
you pull down a new version of the source code, simply run {\bf make setup}
again.


Once Bacula is built, you can run the basic disk only non-root regression test
by entering: 

\footnotesize
\begin{verbatim}
make test
\end{verbatim}
\normalsize


\subsection{Other Tests}
\index{Other Tests}
\index{Tests!Other}
\addcontentsline{toc}{subsection}{Other Tests}

There are a number of other tests that can be run as well. All the tests are a
simply shell script keep in the regress directory. For example the ''make
test`` simply executes {\bf ./all-non-root-tests}. The other tests, which
are invoked by directly running the script are:

\begin{description}

\item [all\_non-root-tests]
   \index{all\_non-root-tests}
   All non-tape tests not requiring root.  This is the standard set of tests,
that in general, backup some  data, then restore it, and finally compares the
restored data  with the original data.  

\item [all-root-tests]
   \index{all-root-tests}
   All non-tape tests requiring root permission.  These are a relatively small
number of tests that require running  as root. The amount of data backed up
can be quite large. For  example, one test backs up /usr, another backs up
/etc. One  or more of these tests reports an error -- I'll fix it one  day. 

\item [all-non-root-tape-tests]
   \index{all-non-root-tape-tests}
   All tape test not requiring root.  There are currently three tests, all run
without being root,  and backup to a tape. The first two tests use one volume,
and the third test requires an autochanger, and uses two  volumes. If you
don't have an autochanger, then this script  will probably produce an error. 

\item [all-tape-and-file-tests]
   \index{all-tape-and-file-tests}
   All tape and file tests not requiring  root. This includes just about
everything, and I don't run it  very often. 
\end{description}

\subsection{If a Test Fails}
\index{Fails!If a Test}
\index{If a Test Fails}
\addcontentsline{toc}{subsection}{If a Test Fails}

If you one or more tests fail, the line output will be similar to: 

\footnotesize
\begin{verbatim}
  !!!!! concurrent-jobs-test failed!!! !!!!!
\end{verbatim}
\normalsize

If you want to determine why the test failed, you will need to rerun the
script with the debug output turned on.  You do so by defining the
environment variable {\bf REGRESS\_DEBUG} with commands such as:

\begin{verbatim}
REGRESS_DEBUG=1
export REGRESS_DEBUG
\end{verbatim}

Then from the "regress" directory (all regression scripts assume that
you have "regress" as the current directory), enter:

\begin{verbatim}
tests/test-name
\end{verbatim}

where test-name should be the name of a test script -- for example:
{\bf tests/backup-bacula-test}.

\section{Testing a Binary Installation}
\index{Test!Testing a Binary Installation}

If you have installed your Bacula from a binary release such as (rpms or debs),
you can still run regression tests on it.
First, make sure that your regression {\bf config} file uses the same catalog backend as
your installed binaries. Then define the variables \texttt{bin} and \texttt{scripts} variables
in your config file.

Example:
\begin{verbatim}
bin=/opt/bacula/bin
scripts=/opt/bacula/scripts
\end{verbatim}

The \texttt{./scripts/prepare-other-loc} will tweak the regress scripts to use
your binary location. You will need to run it manually once before you run any
regression tests.

\begin{verbatim}
$ ./scripts/prepare-other-loc
$ ./tests/backup-bacula-test
...
\end{verbatim}

All regression scripts must be run by hand or by calling the test scripts.  
These are principally scripts that begin with {\bf all\_...} such as {\bf all\_disk\_tests},
{\bf ./all\_test} ...
None of the 
{\bf ./do\_disk}, {\bf ./do\_all}, {\bf ./nightly...}  scripts will work.

If you want to switch back to running the regression scripts from source, first
remove the {\bf bin} and {\bf scripts} variables from your {\bf config} file and
rerun the {\bf make setup} step.

\section{Running a Single Test}
\index{Running a Single Test}
\addcontentsline{toc}{section}{Running a Single Test}

If you wish to run a single test, you can simply:

\begin{verbatim}
cd regress
tests/<name-of-test>
\end{verbatim}

or, if the source code has been updated, you would do:

\begin{verbatim}
cd bacula
git pull
cd regress
make setup
tests/backup-to-null
\end{verbatim}


\section{Writing a Regression Test}
\index{Test!Writing a Regression}
\index{Writing a Regression Test}
\addcontentsline{toc}{section}{Writing a Regression Test}

Any developer, who implements a major new feature, should write a regression
test that exercises and validates the new feature. Each regression test is a
complete test by itself. It terminates any running Bacula, initializes the
database, starts Bacula, then runs the test by using the console program. 

\subsection{Running the Tests by Hand}
\index{Hand!Running the Tests by}
\index{Running the Tests by Hand}
\addcontentsline{toc}{subsection}{Running the Tests by Hand}

You can run any individual test by hand by cd'ing to the {\bf regress}
directory and entering: 

\footnotesize
\begin{verbatim}
tests/<test-name>
\end{verbatim}
\normalsize

\subsection{Directory Structure}
\index{Structure!Directory}
\index{Directory Structure}
\addcontentsline{toc}{subsection}{Directory Structure}

The directory structure of the regression tests is: 

\footnotesize
\begin{verbatim}
  regress                - Makefile, scripts to start tests
    |------ scripts      - Scripts and conf files
    |-------tests        - All test scripts are here
    |
    |------------------ -- All directories below this point are used
    |                       for testing, but are created from the
    |                       above directories and are removed with
    |                       "make distclean"
    |
    |------ bin          - This is the install directory for
    |                        Bacula to be used testing
    |------ build        - Where the Bacula source build tree is
    |------ tmp          - Most temp files go here
    |------ working      - Bacula working directory
    |------ weird-files  - Weird files used in two of the tests.
\end{verbatim}
\normalsize

\subsection{Adding a New Test}
\index{Adding a New Test}
\index{Test!Adding a New}
\addcontentsline{toc}{subsection}{Adding a New Test}

If you want to write a new regression test, it is best to start with one of
the existing test scripts, and modify it to do the new test. 

When adding a new test, be extremely careful about adding anything to any of
the daemons' configuration files. The reason is that it may change the prompts
that are sent to the console. For example, adding a Pool means that the
current scripts, which assume that Bacula automatically selects a Pool, will
now be presented with a new prompt, so the test will fail. If you need to
enhance the configuration files, consider making your own versions. 

\subsection{Running a Test Under The Debugger}
\index{Debugger}
\addcontentsline{toc}{subsection}{Running a Test Under The Debugger}
You can run a test under the debugger (actually run a Bacula daemon
under the debugger) by first setting the environment variable
{\bf REGRESS\_WAIT} with commands such as:

\begin{verbatim}
REGRESS_WAIT=1
export REGRESS_WAIT
\end{verbatim}

Then executing the script.  When the script prints the following line:

\begin{verbatim}
Start Bacula under debugger and enter anything when ready ...
\end{verbatim}

You start the Bacula component you want to run under the debugger in a
different shell window.  For example:

\begin{verbatim}
cd .../regress/bin
gdb bacula-sd 
(possibly set breakpoints, ...)
run -s -f
\end{verbatim}

Then enter any character in the window with the above message. 
An error message will appear saying that the daemon you are debugging
is already running, which is the case. You can simply ignore the
error message.