4 \chapter{Bacula Regression Testing}
6 \index{Testing!Bacula Regression}
7 \index{Bacula Regression Testing}
8 \addcontentsline{toc}{section}{Bacula Regression Testing}
10 \section{Setting up Regession Testing}
11 \index{Setting up Regession Testing}
12 \addcontentsline{toc}{section}{Setting up Regression Testing}
14 This document is intended mostly for developers who wish to ensure that their
15 changes to Bacula don't introduce bugs in the base code. However, you
16 don't need to be a developer to run the regression scripts, and we
17 recommend them before putting your system into production, and before each
18 upgrade, especially if you build from source code. They are
19 simply shell scripts that drive Bacula through bconsole and then typically
20 compare the input and output with {\bf diff}.
22 You can find the existing regression scripts in the Bacula developer's
23 {\bf git} repository on SourceForge. We strongly recommend that you {\bf
24 clone} the repository because afterwards, you can easily get pull the
25 updates that have been made.
27 To get started, we recommend that you create a directory named {\bf
28 bacula}, under which you will put the current source code and the current
29 set of regression scripts. Below, we will describe how to set this up.
31 The top level directory that we call {\bf bacula} can be named anything you
32 want. Note, all the standard regression scripts run as non-root and can be
33 run on the same machine as a production Bacula system (the developers run
36 To create the directory structure for the current trunk and to
37 clone the repository, do the following (note, we assume you
38 are working in your home directory in a non-root account):
42 git clone http://git.bacula.org/bacula bacula
46 This will create the directory {\bf bacula} and populate it with
47 three directories: {\bf bacula}, {\bf gui}, and {\bf regress}.
48 {\bf bacula} contains the Bacula source code; {\bf gui} contains
49 certain gui programs that you will not need, and {\bf regress} contains
50 all the regression scripts. The above should be needed only
51 once. Thereafter to update to the latest code, you do:
60 If you want to test with SQLite and it is not installed on your system,
61 you will need to download the latest depkgs release from Source Forge and
62 unpack it into {\bf depkgs}, then simply:
72 There are two different aspects of regression testing that this document will
73 discuss: 1. Running the Regression Script, 2. Writing a Regression test.
75 \section{Running the Regression Script}
76 \index{Running the Regression Script}
77 \index{Script!Running the Regression}
78 \addcontentsline{toc}{section}{Running the Regression Script}
80 There are a number of different tests that may be run, such as: the standard
81 set that uses disk Volumes and runs under any userid; a small set of tests
82 that write to tape; another set of tests where you must be root to run them.
83 Normally, I run all my tests as non-root and very rarely run the root
84 tests. The tests vary in length, and running the full tests including disk
85 based testing, tape based testing, autochanger based testing, and multiple
86 drive autochanger based testing can take 3 or 4 hours.
88 \subsection{Setting the Configuration Parameters}
89 \index{Setting the Configuration Parameters}
90 \index{Parameters!Setting the Configuration}
91 \addcontentsline{toc}{subsection}{Setting the Configuration Parameters}
93 There is nothing you need to change in the source directory.
105 very first time you are going to run the regression scripts, you will
106 need to create a custom config file for your system.
107 We suggest that you start by:
111 cp prototype.conf config
115 Then you can edit the {\bf config} file directly.
120 # Where to get the source to be tested
121 BACULA_SOURCE="${HOME}/bacula/bacula"
123 # Where to send email !!!!! Change me !!!!!!!
124 EMAIL=your-name@your-domain.com
125 SMTP_HOST="localhost"
127 # Full "default" path where to find sqlite (no quotes!)
128 SQLITE3_DIR=${HOME}/depkgs/sqlite3
129 SQLITE_DIR=${HOME}/depkgs/sqlite
131 TAPE_DRIVE="/dev/nst0"
132 # if you don't have an autochanger set AUTOCHANGER to /dev/null
133 AUTOCHANGER="/dev/sg0"
134 # For two drive tests -- set to /dev/null if you do not have it
135 TAPE_DRIVE1="/dev/null"
137 # This must be the path to the autochanger including its name
138 AUTOCHANGER_PATH="/usr/sbin/mtx"
140 # Set what backend to use "postresql" "mysql" or "sqlite3"
143 # Set your database here
144 #WHICHDB="--with-${DBTYPE}=${SQLITE3_DIR}"
145 WHICHDB="--with-${DBTYPE}"
147 # Set this to "--with-tcp-wrappers" or "--without-tcp-wrappers"
148 TCPWRAPPERS="--with-tcp-wrappers"
150 # Set this to "" to disable OpenSSL support, "--with-openssl=yes"
151 # to enable it, or provide the path to the OpenSSL installation,
152 # eg "--with-openssl=/usr/local"
153 OPENSSL="--with-openssl"
155 # You may put your real host name here, but localhost or 127.0.0.1
156 # is valid also and it has the advantage that it works on a
157 # non-networked machine
164 \item {\bf BACULA\_SOURCE} should be the full path to the Bacula source code
165 that you wish to test. It will be loaded configured, compiled, and
166 installed with the "make setup" command, which needs to be done only
167 once each time you change the source code.
169 \item {\bf EMAIL} should be your email addres. Please remember to change this
170 or I will get a flood of unwanted messages. You may or may not want to see
171 these emails. In my case, I don't need them so I direct it to the bit bucket.
173 \item {\bf SMTP\_HOST} defines where your SMTP server is.
175 \item {\bf SQLITE\_DIR} should be the full path to the sqlite package, must
176 be build before running a Bacula regression, if you are using SQLite. This
177 variable is ignored if you are using MySQL or PostgreSQL. To use PostgreSQL,
178 edit the Makefile and change (or add) WHICHDB?=``\lstinline+--with-postgresql+''. For
179 MySQL use ``WHICHDB=''\lstinline+--with-mysql+``.
181 The advantage of using SQLite is that it is totally independent of any
182 installation you may have running on your system, and there is no
183 special configuration or authorization that must be done to run it.
184 With both MySQL and PostgreSQL, you must pre-install the packages,
185 initialize them and ensure that you have authorization to access the
186 database and create and delete tables.
188 \item {\bf TAPE\_DRIVE} is the full path to your tape drive. The base set of
189 regression tests do not use a tape, so this is only important if you want to
190 run the full tests. Set this to /dev/null if you do not have a tape drive.
192 \item {\bf TAPE\_DRIVE1} is the full path to your second tape drive, if
193 have one. The base set of
194 regression tests do not use a tape, so this is only important if you want to
195 run the full two drive tests. Set this to /dev/null if you do not have a
198 \item {\bf AUTOCHANGER} is the name of your autochanger control device. Set this to
199 /dev/null if you do not have an autochanger.
201 \item {\bf AUTOCHANGER\_PATH} is the full path including the program name for
202 your autochanger program (normally {\bf mtx}. Leave the default value if you
205 \item {\bf TCPWRAPPERS} defines whether or not you want the ./configure
206 to be performed with tcpwrappers enabled.
208 \item {\bf OPENSSL} used to enable/disable SSL support for Bacula
209 communications and data encryption.
211 \item {\bf HOST} is the hostname that it will use when building the
212 scripts. The Bacula daemons will be named <HOST>-dir, <HOST>-fd,
213 ... It is also the name of the HOST machine that to connect to the
214 daemons by the network. Hence the name should either be your real
215 hostname (with an appropriate DNS or /etc/hosts entry) or {\bf
216 localhost} as it is in the default file.
218 \item {\bf bin} is the binary location.
220 \item {\bf scripts} is the bacula scripts location (where we could find
221 database creation script, autochanger handler, etc.)
225 \subsection{Building the Test Bacula}
226 \index{Building the Test Bacula}
227 \index{Bacula!Building the Test}
228 \addcontentsline{toc}{subsection}{Building the Test Bacula}
230 Once the above variables are set, you can build the setup by entering:
238 This will setup the regression testing and you should not need to
239 do this again unless you want to change the database or other regression
240 configuration parameters.
243 \subsection{Setting up your SQL engine}
244 \index{Setting up your SQL engine}
245 \addcontentsline{toc}{subsection}{Setting up your SQL engine}
246 If you are using SQLite or SQLite3, there is nothing more to do; you can
247 simply run the tests as described in the next section.
249 If you are using MySQL or PostgreSQL, you will need to establish an
250 account with your database engine for the user name {\bf regress} and
251 you will need to manually create a database named {\bf regress} that can be
252 used by user name regress, which means you will have to give the user
253 regress sufficient permissions to use the database named regress.
254 There is no password on the regress account.
256 You have probably already done this procedure for the user name and
257 database named bacula. If not, the manual describes roughly how to
258 do it, and the scripts in bacula/regress/build/src/cats named
259 create\_mysql\_database, create\_postgresql\_database, grant\_mysql\_privileges,
260 and grant\_postgresql\_privileges may be of a help to you.
262 Generally, to do the above, you will need to run under root to
263 be able to create databases and modify permissions within MySQL and
266 It is possible to configure MySQL access for database accounts that
267 require a password to be supplied. This can be done by creating a ~/.my.cnf
268 file which supplies the credentials by default to the MySQL commandline
278 A similar technique can be used PostgreSQL regression testing where the
279 database is configured to require a password. The ~/.pgpass file should
280 contain a line with the database connection properties.
283 hostname:port:database:username:password
286 \subsection{Running the Disk Only Regression}
287 \index{Regression!Running the Disk Only}
288 \index{Running the Disk Only Regression}
289 \addcontentsline{toc}{subsection}{Running the Disk Only Regression}
291 The simplest way to copy the source code, configure it, compile it, link
292 it, and run the tests is to use a helper script:
303 This will run the base set of tests using disk Volumes.
304 If you are testing on a
305 non-Linux machine several of the of the tests may not be run. In any case,
306 as we add new tests, the number will vary. It will take about 1 hour
307 and you don't need to be root
308 to run these tests (I run under my regular userid). The result should be
309 something similar to:
314 ===== auto-label-test OK 12:31:33 =====
315 ===== backup-bacula-test OK 12:32:32 =====
316 ===== bextract-test OK 12:33:27 =====
317 ===== bscan-test OK 12:34:47 =====
318 ===== bsr-opt-test OK 12:35:46 =====
319 ===== compressed-test OK 12:36:52 =====
320 ===== compressed-encrypt-test OK 12:38:18 =====
321 ===== concurrent-jobs-test OK 12:39:49 =====
322 ===== data-encrypt-test OK 12:41:11 =====
323 ===== encrypt-bug-test OK 12:42:00 =====
324 ===== fifo-test OK 12:43:46 =====
325 ===== backup-bacula-fifo OK 12:44:54 =====
326 ===== differential-test OK 12:45:36 =====
327 ===== four-concurrent-jobs-test OK 12:47:39 =====
328 ===== four-jobs-test OK 12:49:22 =====
329 ===== incremental-test OK 12:50:38 =====
330 ===== query-test OK 12:51:37 =====
331 ===== recycle-test OK 12:53:52 =====
332 ===== restore2-by-file-test OK 12:54:53 =====
333 ===== restore-by-file-test OK 12:55:40 =====
334 ===== restore-disk-seek-test OK 12:56:29 =====
335 ===== six-vol-test OK 12:57:44 =====
336 ===== span-vol-test OK 12:58:52 =====
337 ===== sparse-compressed-test OK 13:00:00 =====
338 ===== sparse-test OK 13:01:04 =====
339 ===== two-jobs-test OK 13:02:39 =====
340 ===== two-vol-test OK 13:03:49 =====
341 ===== verify-vol-test OK 13:04:56 =====
342 ===== weird-files2-test OK 13:05:47 =====
343 ===== weird-files-test OK 13:06:33 =====
344 ===== migration-job-test OK 13:08:15 =====
345 ===== migration-jobspan-test OK 13:09:33 =====
346 ===== migration-volume-test OK 13:10:48 =====
347 ===== migration-time-test OK 13:12:59 =====
348 ===== hardlink-test OK 13:13:50 =====
349 ===== two-pool-test OK 13:18:17 =====
350 ===== fast-two-pool-test OK 13:24:02 =====
351 ===== two-volume-test OK 13:25:06 =====
352 ===== incremental-2disk OK 13:25:57 =====
353 ===== 2drive-incremental-2disk OK 13:26:53 =====
354 ===== scratch-pool-test OK 13:28:01 =====
355 Total time = 0:57:55 or 3475 secs
360 and the working tape tests are run with
373 ===== Bacula tape test OK =====
374 ===== Small File Size test OK =====
375 ===== restore-by-file-tape test OK =====
376 ===== incremental-tape test OK =====
377 ===== four-concurrent-jobs-tape OK =====
378 ===== four-jobs-tape OK =====
382 Each separate test is self contained in that it initializes to run Bacula from
383 scratch (i.e. newly created database). It will also kill any Bacula session
384 that is currently running. In addition, it uses ports 8101, 8102, and 8103 so
385 that it does not intefere with a production system.
387 Alternatively, you can do the ./do\_disk work by hand with:
395 The above will then copy the source code within
396 the regression tree (in directory regress/build), configure it, and build it.
397 There should be no errors. If there are, please correct them before
398 continuing. From this point on, as long as you don't change the Bacula
399 source code, you should not need to repeat any of the above steps. If
400 you pull down a new version of the source code, simply run {\bf make setup}
404 Once Bacula is built, you can run the basic disk only non-root regression test
414 \subsection{Other Tests}
417 \addcontentsline{toc}{subsection}{Other Tests}
419 There are a number of other tests that can be run as well. All the tests are a
420 simply shell script keep in the regress directory. For example the ''make
421 test`` simply executes {\bf ./all-non-root-tests}. The other tests, which
422 are invoked by directly running the script are:
426 \item [all\_non-root-tests]
427 \index{all\_non-root-tests}
428 All non-tape tests not requiring root. This is the standard set of tests,
429 that in general, backup some data, then restore it, and finally compares the
430 restored data with the original data.
432 \item [all-root-tests]
433 \index{all-root-tests}
434 All non-tape tests requiring root permission. These are a relatively small
435 number of tests that require running as root. The amount of data backed up
436 can be quite large. For example, one test backs up /usr, another backs up
437 /etc. One or more of these tests reports an error -- I'll fix it one day.
439 \item [all-non-root-tape-tests]
440 \index{all-non-root-tape-tests}
441 All tape test not requiring root. There are currently three tests, all run
442 without being root, and backup to a tape. The first two tests use one volume,
443 and the third test requires an autochanger, and uses two volumes. If you
444 don't have an autochanger, then this script will probably produce an error.
446 \item [all-tape-and-file-tests]
447 \index{all-tape-and-file-tests}
448 All tape and file tests not requiring root. This includes just about
449 everything, and I don't run it very often.
452 \subsection{If a Test Fails}
453 \index{Fails!If a Test}
454 \index{If a Test Fails}
455 \addcontentsline{toc}{subsection}{If a Test Fails}
457 If you one or more tests fail, the line output will be similar to:
461 !!!!! concurrent-jobs-test failed!!! !!!!!
465 If you want to determine why the test failed, you will need to rerun the
466 script with the debug output turned on. You do so by defining the
467 environment variable {\bf REGRESS\_DEBUG} with commands such as:
474 Then from the "regress" directory (all regression scripts assume that
475 you have "regress" as the current directory), enter:
481 where test-name should be the name of a test script -- for example:
482 {\bf tests/backup-bacula-test}.
484 \section{Testing a Binary Installation}
485 \index{Test!Testing a Binary Installation}
487 If you have installed your Bacula from a binary release such as (rpms or debs),
488 you can still run regression tests on it.
489 First, make sure that your regression {\bf config} file uses the same catalog backend as
490 your installed binaries. Then define the variables \texttt{bin} and \texttt{scripts} variables
496 scripts=/opt/bacula/scripts
499 The \texttt{./scripts/prepare-other-loc} will tweak the regress scripts to use
500 your binary location. You will need to run it manually once before you run any
504 $ ./scripts/prepare-other-loc
505 $ ./tests/backup-bacula-test
509 All regression scripts must be run by hand or by calling the test scripts.
510 These are principally scripts that begin with {\bf all\_...} such as {\bf all\_disk\_tests},
511 {\bf ./all\_test} ...
513 {\bf ./do\_disk}, {\bf ./do\_all}, {\bf ./nightly...} scripts will work.
515 If you want to switch back to running the regression scripts from source, first
516 remove the {\bf bin} and {\bf scripts} variables from your {\bf config} file and
517 rerun the {\bf make setup} step.
519 \section{Running a Single Test}
520 \index{Running a Single Test}
521 \addcontentsline{toc}{section}{Running a Single Test}
523 If you wish to run a single test, you can simply:
530 or, if the source code has been updated, you would do:
541 \section{Writing a Regression Test}
542 \index{Test!Writing a Regression}
543 \index{Writing a Regression Test}
544 \addcontentsline{toc}{section}{Writing a Regression Test}
546 Any developer, who implements a major new feature, should write a regression
547 test that exercises and validates the new feature. Each regression test is a
548 complete test by itself. It terminates any running Bacula, initializes the
549 database, starts Bacula, then runs the test by using the console program.
551 \subsection{Running the Tests by Hand}
552 \index{Hand!Running the Tests by}
553 \index{Running the Tests by Hand}
554 \addcontentsline{toc}{subsection}{Running the Tests by Hand}
556 You can run any individual test by hand by cd'ing to the {\bf regress}
557 directory and entering:
565 \subsection{Directory Structure}
566 \index{Structure!Directory}
567 \index{Directory Structure}
568 \addcontentsline{toc}{subsection}{Directory Structure}
570 The directory structure of the regression tests is:
574 regress - Makefile, scripts to start tests
575 |------ scripts - Scripts and conf files
576 |-------tests - All test scripts are here
578 |------------------ -- All directories below this point are used
579 | for testing, but are created from the
580 | above directories and are removed with
583 |------ bin - This is the install directory for
584 | Bacula to be used testing
585 |------ build - Where the Bacula source build tree is
586 |------ tmp - Most temp files go here
587 |------ working - Bacula working directory
588 |------ weird-files - Weird files used in two of the tests.
592 \subsection{Adding a New Test}
593 \index{Adding a New Test}
594 \index{Test!Adding a New}
595 \addcontentsline{toc}{subsection}{Adding a New Test}
597 If you want to write a new regression test, it is best to start with one of
598 the existing test scripts, and modify it to do the new test.
600 When adding a new test, be extremely careful about adding anything to any of
601 the daemons' configuration files. The reason is that it may change the prompts
602 that are sent to the console. For example, adding a Pool means that the
603 current scripts, which assume that Bacula automatically selects a Pool, will
604 now be presented with a new prompt, so the test will fail. If you need to
605 enhance the configuration files, consider making your own versions.
607 \subsection{Running a Test Under The Debugger}
609 \addcontentsline{toc}{subsection}{Running a Test Under The Debugger}
610 You can run a test under the debugger (actually run a Bacula daemon
611 under the debugger) by first setting the environment variable
612 {\bf REGRESS\_WAIT} with commands such as:
619 Then executing the script. When the script prints the following line:
622 Start Bacula under debugger and enter anything when ready ...
625 You start the Bacula component you want to run under the debugger in a
626 different shell window. For example:
631 (possibly set breakpoints, ...)
635 Then enter any character in the window with the above message.
636 An error message will appear saying that the daemon you are debugging
637 is already running, which is the case. You can simply ignore the