4 \chapter{What To Do When Bacula Crashes (Kaboom)}
6 \index[general]{Kaboom!What To Do When Bacula Crashes }
7 \index[general]{What To Do When Bacula Crashes (Kaboom) }
9 If you are running on a Linux system, and you have a set of working
10 configuration files, it is very unlikely that {\bf Bacula} will crash. As with
11 all software, however, it is inevitable that someday, it may crash,
12 particularly if you are running on another operating system or using a new or
15 This chapter explains what you should do if one of the three {\bf Bacula}
16 daemons (Director, File, Storage) crashes. When we speak of crashing, we
17 mean that the daemon terminates abnormally because of an error. There are
18 many cases where Bacula detects errors (such as PIPE errors) and will fail
19 a job. These are not considered crashes. In addition, under certain
20 conditions, Bacula will detect a fatal in the configuration, such as
21 lack of permission to read/write the working directory. In that case,
22 Bacula will force itself to crash with a SEGFAULT. However, before
23 crashing, Bacula will normally display a message indicating why.
24 For more details, please read on.
28 \index[general]{Traceback }
30 Each of the three Bacula daemons has a built-in exception handler which, in
31 case of an error, will attempt to produce a traceback. If successful the
32 traceback will be emailed to you.
34 For this to work, you need to ensure that a few things are setup correctly on
38 \item You must have an installed copy of {\bf gdb} (the GNU debugger), and it
39 must be on {\bf Bacula's} path. On some systems such as Solaris, {\bf
40 gdb} may be replaced by {\bf dbx}.
41 \item The Bacula installed script file {\bf btraceback} must be in the same
42 directory as the daemon which dies, and it must be marked as executable.
43 \item The script file {\bf btraceback.gdb} must have the correct path to it
44 specified in the {\bf btraceback} file.
45 \item You must have a {\bf mail} program which is on {\bf Bacula's} path.
46 By default, this {\bf mail} program is set to {\bf bsmtp}, so it must
47 be correctly configured.
50 If all the above conditions are met, the daemon that crashes will produce a
51 traceback report and email it to you. If the above conditions are not true,
52 you can either run the debugger by hand as described below, or you may be able
53 to correct the problems by editing the {\bf btraceback} file. I recommend not
54 spending too much time on trying to get the traceback to work as it can be
57 The changes that might be needed are to add a correct path to the {\bf gdb}
58 program, correct the path to the {\bf btraceback.gdb} file, change the {\bf
59 mail} program or its path, or change your email address. The key line in the
60 {\bf btraceback} file is:
64 gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \
65 $1 $2 2>\&1 | bsmtp -s "Bacula traceback" your-address@xxx.com
69 Since each daemon has the same traceback code, a single btraceback file is
70 sufficient if you are running more than one daemon on a machine.
72 \section{Testing The Traceback}
73 \index[general]{Traceback!Testing The }
74 \index[general]{Testing The Traceback }
76 To "manually" test the traceback feature, you simply start {\bf Bacula} then
77 obtain the {\bf PID} of the main daemon thread (there are multiple threads).
78 The output produced here will look different depending on what OS and what
79 version of the kernel you are running.
80 Unfortunately, the output had to be split to fit on this page:
84 [kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir
85 2103 ? S 0:00 /home/kern/bacula/k/src/dird/bacula-dir -c
86 /home/kern/bacula/k/src/dird/dird.conf
87 2104 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
88 /home/kern/bacula/k/src/dird/dird.conf
89 2106 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
90 /home/kern/bacula/k/src/dird/dird.conf
91 2105 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
92 /home/kern/bacula/k/src/dird/dird.conf
96 which in this case is 2103. Then while Bacula is running, you call the program
97 giving it the path to the Bacula executable and the {\bf PID}. In this case,
102 ./btraceback /home/kern/bacula/k/src/dird 2103
106 It should produce an email showing you the current state of the daemon (in
107 this case the Director), and then exit leaving {\bf Bacula} running as if
108 nothing happened. If this is not the case, you will need to correct the
109 problem by modifying the {\bf btraceback} script.
111 Typical problems might be that {\bf gdb} or {\bf dbx} for Solaris is not on
112 the default path. Fix this by specifying the full path to it in the {\bf
113 btraceback} file. Another common problem is that you haven't modified the
114 script so that the {\bf bsmtp} program has an appropriate smtp server or
115 the proper syntax for your smtp server. If you use the {\bf mail} program
116 and it is not on the default path, it will also fail. On some systems, it
117 is preferable to use {\bf Mail} rather than {\bf mail}.
119 \section{Getting A Traceback On Other Systems}
120 \index[general]{Getting A Traceback On Other Systems }
121 \index[general]{Systems!Getting A Traceback On Other }
123 It should be possible to produce a similar traceback on systems other than
124 Linux, either using {\bf gdb} or some other debugger. Solaris with {\bf dbx}
125 loaded works quite fine. On other systems, you will need to modify the {\bf
126 btraceback} program to invoke the correct debugger, and possibly correct the
127 {\bf btraceback.gdb} script to have appropriate commands for your debugger. If
128 anyone succeeds in making this work with another debugger, please send us a
129 copy of what you modified.
130 \label{ManuallyDebugging}
132 \section{Manually Running Bacula Under The Debugger}
133 \index[general]{Manually Running Bacula Under The Debugger }
134 \index[general]{Debugger!Manually Running Bacula Under The }
136 If for some reason you cannot get the automatic traceback, or if you want to
137 interactively examine the variable contents after a crash, you can run Bacula
138 under the debugger. Assuming you want to run the Storage daemon under the
139 debugger (the technique is the same for the other daemons, only the name
140 changes), you would do the following:
143 \item Start the Director and the File daemon. If the Storage daemon also
144 starts, you will need to find its PID as shown above (ps fax | grep
145 bacula-sd) and kill it with a command like the following:
154 where you replace {\bf PID} by the actual value.
155 \item At this point, the Director and the File daemon should be running but
156 the Storage daemon should not.
157 \item cd to the directory containing the Storage daemon
158 \item Start the Storage daemon under the debugger:
167 \item Run the Storage daemon:
171 run -s -f -c ./bacula-sd.conf
176 You may replace the {\bf ./bacula-sd.conf} with the full path to the Storage
177 daemon's configuration file.
178 \item At this point, Bacula will be fully operational.
179 \item In another shell command window, start the Console program and do what
180 is necessary to cause Bacula to die.
181 \item When Bacula crashes, the {\bf gdb} shell window will become active and
182 {\bf gdb} will show you the error that occurred.
183 \item To get a general traceback of all threads, issue the following command:
193 After that you can issue any debugging command.
196 \section{Getting Debug Output from Bacula}
197 \index[general]{Getting Debug Output from Bacula }
199 Each of the daemons normally has debug compiled into the program, but
200 disabled. There are two ways to enable the debug output. One is to add the
201 {\bf -d nnn} option on the command line when starting the debugger. The {\bf
202 nnn} is the debug level, and generally anything between 50 and 200 is
203 reasonable. The higher the number, the more output is produced. The output is
204 written to standard output.
206 The second way of getting debug output is to dynamically turn it on using the
207 Console using the {\bf setdebug} command. The full syntax of the command is:
211 setdebug level=nnn client=client-name storage=storage-name dir
215 If none of the options are given, the command will prompt you. You can
216 selectively turn on/off debugging in any or all the daemons (i.e. it is not
217 necessary to specify all the components of the above command).