--- /dev/null
+%%
+%%
+
+\chapter{What To Do When Bacula Crashes (Kaboom)}
+\label{KaboomChapter}
+\index[general]{Kaboom!What To Do When Bacula Crashes }
+\index[general]{What To Do When Bacula Crashes (Kaboom) }
+
+If you are running on a Linux system, and you have a set of working
+configuration files, it is very unlikely that {\bf Bacula} will crash. As with
+all software, however, it is inevitable that someday, it may crash,
+particularly if you are running on another operating system or using a new or
+unusual feature.
+
+This chapter explains what you should do if one of the three {\bf Bacula}
+daemons (Director, File, Storage) crashes. When we speak of crashing, we
+mean that the daemon terminates abnormally because of an error. There are
+many cases where Bacula detects errors (such as PIPE errors) and will fail
+a job. These are not considered crashes. In addition, under certain
+conditions, Bacula will detect a fatal in the configuration, such as
+lack of permission to read/write the working directory. In that case,
+Bacula will force itself to crash with a SEGFAULT. However, before
+crashing, Bacula will normally display a message indicating why.
+For more details, please read on.
+
+
+\section{Traceback}
+\index[general]{Traceback}
+
+Each of the three Bacula daemons has a built-in exception handler which, in
+case of an error, will attempt to produce a traceback. If successful the
+traceback will be emailed to you.
+
+For this to work, you need to ensure that a few things are setup correctly on
+your system:
+
+\begin{enumerate}
+\item You must have a version of Bacula built with debug information turned
+ on and not stripped of debugging symbols.
+
+\item You must have an installed copy of {\bf gdb} (the GNU debugger), and it
+ must be on {\bf Bacula's} path. On some systems such as Solaris, {\bf
+ gdb} may be replaced by {\bf dbx}.
+
+\item The Bacula installed script file {\bf btraceback} must be in the same
+ directory as the daemon which dies, and it must be marked as executable.
+
+\item The script file {\bf btraceback.gdb} must have the correct path to it
+ specified in the {\bf btraceback} file.
+
+\item You must have a {\bf mail} program which is on {\bf Bacula's} path.
+ By default, this {\bf mail} program is set to {\bf bsmtp}, so it must
+ be correctly configured.
+
+\item If you run either the Director or Storage daemon under a non-root
+ userid, you will most likely need to modify the {\bf btraceback} file
+ to do something like {\bf sudo} (raise to root priority) for the
+ call to {\bf gdb} so that it has the proper permissions to debug
+ Bacula.
+\end{enumerate}
+
+If all the above conditions are met, the daemon that crashes will produce a
+traceback report and email it to you. If the above conditions are not true,
+you can either run the debugger by hand as described below, or you may be able
+to correct the problems by editing the {\bf btraceback} file. I recommend not
+spending too much time on trying to get the traceback to work as it can be
+very difficult.
+
+The changes that might be needed are to add a correct path to the {\bf gdb}
+program, correct the path to the {\bf btraceback.gdb} file, change the {\bf
+mail} program or its path, or change your email address. The key line in the
+{\bf btraceback} file is:
+
+\footnotesize
+\begin{verbatim}
+gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \
+ $1 $2 2>\&1 | bsmtp -s "Bacula traceback" your-address@xxx.com
+\end{verbatim}
+\normalsize
+
+Since each daemon has the same traceback code, a single btraceback file is
+sufficient if you are running more than one daemon on a machine.
+
+\section{Testing The Traceback}
+\index[general]{Traceback!Testing The }
+\index[general]{Testing The Traceback }
+
+To "manually" test the traceback feature, you simply start {\bf Bacula} then
+obtain the {\bf PID} of the main daemon thread (there are multiple threads).
+The output produced here will look different depending on what OS and what
+version of the kernel you are running.
+Unfortunately, the output had to be split to fit on this page:
+
+\footnotesize
+\begin{verbatim}
+[kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir
+ 2103 ? S 0:00 /home/kern/bacula/k/src/dird/bacula-dir -c
+ /home/kern/bacula/k/src/dird/dird.conf
+ 2104 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
+ /home/kern/bacula/k/src/dird/dird.conf
+ 2106 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
+ /home/kern/bacula/k/src/dird/dird.conf
+ 2105 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
+ /home/kern/bacula/k/src/dird/dird.conf
+\end{verbatim}
+\normalsize
+
+which in this case is 2103. Then while Bacula is running, you call the program
+giving it the path to the Bacula executable and the {\bf PID}. In this case,
+it is:
+
+\footnotesize
+\begin{verbatim}
+./btraceback /home/kern/bacula/k/src/dird 2103
+\end{verbatim}
+\normalsize
+
+It should produce an email showing you the current state of the daemon (in
+this case the Director), and then exit leaving {\bf Bacula} running as if
+nothing happened. If this is not the case, you will need to correct the
+problem by modifying the {\bf btraceback} script.
+
+Typical problems might be that {\bf gdb} or {\bf dbx} for Solaris is not on
+the default path. Fix this by specifying the full path to it in the {\bf
+btraceback} file. Another common problem is that you haven't modified the
+script so that the {\bf bsmtp} program has an appropriate smtp server or
+the proper syntax for your smtp server. If you use the {\bf mail} program
+and it is not on the default path, it will also fail. On some systems, it
+is preferable to use {\bf Mail} rather than {\bf mail}.
+
+\section{Getting A Traceback On Other Systems}
+\index[general]{Getting A Traceback On Other Systems}
+\index[general]{Systems!Getting A Traceback On Other}
+
+It should be possible to produce a similar traceback on systems other than
+Linux, either using {\bf gdb} or some other debugger. Solaris with {\bf dbx}
+loaded works quite fine. On other systems, you will need to modify the {\bf
+btraceback} program to invoke the correct debugger, and possibly correct the
+{\bf btraceback.gdb} script to have appropriate commands for your debugger. If
+anyone succeeds in making this work with another debugger, please send us a
+copy of what you modified. Please keep in mind that for any debugger to
+work, it will most likely need to run as root, so you may need to modify
+the {\bf btraceback} script accordingly.
+
+\label{ManuallyDebugging}
+\section{Manually Running Bacula Under The Debugger}
+\index[general]{Manually Running Bacula Under The Debugger}
+\index[general]{Debugger!Manually Running Bacula Under The}
+
+If for some reason you cannot get the automatic traceback, or if you want to
+interactively examine the variable contents after a crash, you can run Bacula
+under the debugger. Assuming you want to run the Storage daemon under the
+debugger (the technique is the same for the other daemons, only the name
+changes), you would do the following:
+
+\begin{enumerate}
+\item Start the Director and the File daemon. If the Storage daemon also
+ starts, you will need to find its PID as shown above (ps fax | grep
+ bacula-sd) and kill it with a command like the following:
+
+\footnotesize
+\begin{verbatim}
+ kill -15 PID
+\end{verbatim}
+\normalsize
+
+where you replace {\bf PID} by the actual value.
+
+\item At this point, the Director and the File daemon should be running but
+ the Storage daemon should not.
+
+\item cd to the directory containing the Storage daemon
+
+\item Start the Storage daemon under the debugger:
+
+ \footnotesize
+\begin{verbatim}
+ gdb ./bacula-sd
+\end{verbatim}
+\normalsize
+
+\item Run the Storage daemon:
+
+ \footnotesize
+\begin{verbatim}
+ run -s -f -c ./bacula-sd.conf
+\end{verbatim}
+\normalsize
+
+You may replace the {\bf ./bacula-sd.conf} with the full path to the Storage
+daemon's configuration file.
+
+\item At this point, Bacula will be fully operational.
+
+\item In another shell command window, start the Console program and do what
+ is necessary to cause Bacula to die.
+
+\item When Bacula crashes, the {\bf gdb} shell window will become active and
+ {\bf gdb} will show you the error that occurred.
+
+\item To get a general traceback of all threads, issue the following command:
+
+
+\footnotesize
+\begin{verbatim}
+ thread apply all bt
+\end{verbatim}
+\normalsize
+
+After that you can issue any debugging command.
+\end{enumerate}
+
+\section{Getting Debug Output from Bacula}
+\index[general]{Getting Debug Output from Bacula }
+Each of the daemons normally has debug compiled into the program, but
+disabled. There are two ways to enable the debug output. One is to add the
+{\bf -d nnn} option on the command line when starting the debugger. The {\bf
+nnn} is the debug level, and generally anything between 50 and 200 is
+reasonable. The higher the number, the more output is produced. The output is
+written to standard output.
+
+The second way of getting debug output is to dynamically turn it on using the
+Console using the {\bf setdebug} command. The full syntax of the command is:
+
+\footnotesize
+\begin{verbatim}
+ setdebug level=nnn client=client-name storage=storage-name dir
+\end{verbatim}
+\normalsize
+
+If none of the options are given, the command will prompt you. You can
+selectively turn on/off debugging in any or all the daemons (i.e. it is not
+necessary to specify all the components of the above command).
projects / bacula / docs / blobdiff