Update

[bacula/docs] / docs / developers / generaldevel.tex

%%
%%

\section*{Bacula Developer Notes}
\label{_ChapterStart10}
\index{Bacula Developer Notes}
\index{Notes!Bacula Developer}
\addcontentsline{toc}{section}{Bacula Developer Notes}

\subsection*{General}
\index{General}
\addcontentsline{toc}{subsection}{General}

This document is intended mostly for developers and describes the the general
framework of making Bacula source changes. 

\subsubsection*{Contributions}
\index{Contributions}
\addcontentsline{toc}{subsubsection}{Contributions}

Contributions from programmers are broken into two groups. The first are
contributions that are aids and not essential to Bacula. In general, these
will be scripts or will go into and examples or contributions directory. 
For these kinds of non-essential contributions there is no obligation to do
a copyright assignment as described below. However, a copyright assignment
would still be appreciated.

The second class of contributions are those which will be integrated with
Bacula and become an essential part. Within this class of contributions, there
are two hurdles to surmount. One is getting your patch accepted, and two is
dealing with copyright issues. The following text describes some of the
requirements for such code. 

\subsubsection*{Patches}
\index{Patches}
\addcontentsline{toc}{subsubsection}{Patches}

Subject to the copyright assignment described below, your patches should be
sent in {\bf diff -u} format relative to the current contents of the Source
Forge CVS, which is the easiest to understand and integrate.
Please be sure to use the Bacula indenting standard (see below).
If you have checked out the source with CVS, you can get a diff using:

\begin{verbatim}
cvs diff -u > change.patch
\end{verbatim}
     
If you plan on doing significant development work over a period of time,
after having your first patch reviewed and approved, you will be eligible
for having developer CVS access so that you can commit your changes
directly to the CVS repository.  To do so, you will need a userid on Source
Forge.

\subsubsection*{Copyrights}
\index{Copyrights}
\addcontentsline{toc}{subsubsection}{Copyrights}

To avoid future problems concerning changing licensing or copyrights, all
code contributions more than a hand full of lines must be in the Public
Domain or have the copyright assigned to Kern Sibbald as in the current
code.  Note, prior to November 2004, the code was copyrighted by Kern
Sibbald and John Walker. When some sort of Bacula entity is created, this
this copyright will be transferred to that entity.  Hopefully this will
happen by Q1-2007.

Your name should be clearly indicated as the author of the code, and you
must be extremely careful not to violate any copyrights or use other
people's code without acknowledging it.  The purpose of this requirement is
to avoid future copyright, patent, or intellectual property problems.  To
understand on possible source of future problems, please examine the
difficulties Mozilla is (was?) having finding previous contributors at
\elink{ http://www.mozilla.org/MPL/missing.html}
{http://www.mozilla.org/MPL/missing.html}. The other important issue is to
avoid copyright, patent, or intellectual property violations as are currently
(May 2003) being claimed by SCO against IBM. 

Although the copyright will be held by Kern, each developer is expected to
indicate that he wrote and/or modified a particular module (or file) and
any other sources.  The copyright assignment may seem a bit unusual, but in
reality, it is not.  Most large projects require this.  In fact, the
paperwork associated with making contributions to the Free Software
Foundation, was for me unsurmountable, so hopefully the rather
simplified procedure we have will not create any difficulties for
you.

If you have any doubts about this, please don't hesitate to ask.  The
objective is to assure the long term servival of the Bacula project. 

Items not needing a copyright assignment are: most small changes,
enhancements, or bug fixes of 5-10 lines of code.                    

\subsubsection*{Copyright Assignment}
\index{Copyright Assignment}
\index{Assignment!Copyright}
\addcontentsline{toc}{subsubsection}{Copyright Assignment}

Since this is not a commercial enterprise, and I prefer to believe in
everyone's good faith, previously developers could assign the copyright by
explicitly acknowledging that they do so in their first submission.  This
was sufficient if the developer is independent, or an employee of a
not-for-profit organization or a university.  However, in an effort to
ensure that the Bacula code is really clean, beginning in August 2006, all
previous and future developers with CVS access will be asked to submit a
copyright assignment.

Any developer who wants to contribute and is employed by a company must get
a copyright assignment from his employer.  This is to avoid
misunderstandings between the employee, the company, and the Bacula
project.  A good number of companies have already followed this procedure.

\subsubsection*{Personal Copyright Assignment Statement}
\index{Copyright!Personal Assignment}
\index{Personal Copyright Assignment Statement}
\addcontentsline{toc}{subsubsection}{Personal Copyright Assignment Statement}

The following statement must be filled out by each non-employed developer
(please see below for developer's that are employed), signed, and mailed
to my address (please ask me for my address and I will email it -- I'd prefer
not to include it here). 

\footnotesize
\begin{verbatim}
Copyright release and transfer statement.
   From: <your name>

   To: Kern Sibbald
   Concerning: Copyright release and transfer

   The undersigned hereby certifies that he/she is independent
   and wishes to develop
   code for and contribute code to the Bacula project for an
   undetermined period of time.  The copyright as well as all
   other rights to and interests in such contributed code are
   hereby transferred to Kern Sibbald.

   Signed in <City, Country> on <Date>:

   <Name of Person>

\end{verbatim}
\normalsize

This release/transfer statement must be sent to:
Kern Sibbald
Address-to-be-given

If you wish to retain the full rights to use the software you
have contributed in different projects, this is not a problem. Just
request a perpetual non-exclusive license before sending in your
copyright assignment.  See the file <bacula-source>/src/lib/tls.c
for an example of such a license.


\subsubsection*{Corporate Copyright Assignment Statement}
\index{Copyright!Corporate Assignment}
\index{Corporate Copyright Assignment Statement}
\addcontentsline{toc}{subsubsection}{Corporate Copyright Assignment Statement}

The following statement must be filled out by the employer, signed, and
mailed to my address (please ask me for my address and I will email it --
I'd prefer not to include it here).

\footnotesize
\begin{verbatim}
Copyright release and transfer statement.
   <On company letter head>

   To: Kern Sibbald
   Concerning: Copyright release and transfer

   <Company, Inc> is hereby agrees that <names-of-developers> and
   other employees of <Company, Inc> are authorized to develop
   code for and contribute code to the Bacula project for an
   undetermined period of time.  The copyright as well as all
   other rights to and interests in such contributed code are
   hereby transferred to Kern Sibbald.

   Signed in <City, Country> on <Date>:

   <Name of Person>, <Position in Company, Inc>

\end{verbatim}
\normalsize

This release/transfer statement must be sent to:
Kern Sibbald
Address-to-be-given

If you wish to retain the full rights to use the software you
have contributed in different projects, this is not a problem. Just
request a perpetual non-exclusive license before sending in your
copyright assignment. See the file <bacula-source>/src/lib/tls.c
for an example of such a license.


\subsection*{Basic CVS Usage}
\index{Basic CVS Usage}
\index{CVS}
\addcontentsline{toc}{subsection}{Basic CVS Usage}

The Bacula CVS is kept at Source Forge. If you are not a developer,
you only be able to access the public CVS, which runs about 6 hours behind
the developer's CVS.  If you are a developer, then you will have immediate
access to "the" CVS and any changes ("commit") that you make will be
immediately seen by other developers.

For developer access to the CVS, go to the Bacula page on Source Forge
and click on the CVS menu item.  Then follow the instructions for
installing your SSH public key.  It may take a few hours until the
key is actually installed, and then you will have access to the CVS.

The Bacula CVS is divided into the following CVS "projects" or "modules".

\begin{verbatim}

  bacula              (Bacula source code)
  docs                (Bacula documentation)
  gui                 (Some of the GUI programs that do not use
                       the Bacula core code)
  rescue              (Bacula CDROM rescue code)
  regress             (Bacula regression scripts)
  ryol                (Roll your own Linux -- incomplete project)

\end{verbatim}

Most of you will want either the Bacula source (bacula) and/or the
documents (docs).

To get the source for a project, you must check it out ("checkout"), which
you do usually once.

\subsubsection*{Public CVS Access}
\index{Public CVS Access}
\addcontentsline{toc}{subsubsection}{Public CVS Access}

The first time you checkout the code for each project, you will need to
tell the cvs program where the CVS repository is.  The procedure for
checking out code from the public CVS is:

\begin{verbatim}
cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bacula login 
\end{verbatim}
Then when it prompts for the password for {\bf anonymous}, simply
press the Enter key.  The above command is necessary only once
the very first time you login. Then enter the following command:

\begin{verbatim}
cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bacula co -P bacula
\end{verbatim}

The above will place the contents of the bacula module in a directory
named {\bf bacula} in the current directory.  This data will come
from the public CVS, which typically runs 6 hours to a day behind the
developer's CVS. Once you have created a copy of the CVS, you can use 
the commands listed below under the title CVS Usage.

\subsubsection*{Developer CVS Access}
\index{Developer CVS Access}
\addcontentsline{toc}{subsubsection}{Developer CVS Access}

If you are registered as a Bacula developer (contact Kern about this),
you may access the developer's CVS using:

\begin{verbatim}
export CVS_RSH=ssh
export CVSROOT=:ext:<nnnn>@cvs.bacula.sourceforge.net:/cvsroot/bacula
\end{verbatim}

where you replace \lt{}nnnn\gt{} by your Source Forge user name.

Then do:

\begin{verbatim}
cvs -z3 checkout -d <directory> bacula
\end{verbatim}

where you replace \lt{}directory\gt{} by the name of the directory you want
to contain the Bacula source code.  If you want the docs, replace the
word "bacula" on the above line by "docs" and be sure to put it in
a different directory.  The -z3 just tells CVS to use compression during
the transmission, which makes things go faster. There is no need to
do the anonymous login as is the case for non-developers.

The above command should generate output that looks a bit like the
following:

\begin{verbatim}
cvs checkout: Updating bacula
U bacula/Makefile
U bacula/bar.c
U bacula/foo.c
U bacula/main.c
...
\end{verbatim}

\subsubsection*{CVS Usage}
\index{CVS Usage}
\addcontentsline{toc}{subsubsection}{CVS Usage}

The commands that follow with the exception of the {\bf commit} 
work the same whether you are accessing the public CVS or
the developer's CVS.

Let's assume you used the name "bacula" for the directory, so your
command was:

\begin{verbatim}
cvs -z3 checkout -d bacula bacula
\end{verbatim}

When the command is done, simply do:

\begin{verbatim}
cd bacula
\end{verbatim}


and then build Bacula.  You will notice a lot of extra CVS directories
in the source code.  Don't ever change or delete them, or you will mess up
your copy of the project. Also, do not rename or delete any of the files
in your copy of the repository or you may have problems.  Any files that
you change will remain only on your system until you do a "commit", which
is explained later.

Let's say you are working in the directory scripts.  You would then do:

\begin{verbatim}
cd scripts
(edit some files)
\end{verbatim}

when you are happy with your changes, you can do the following:

\begin{verbatim}
cd bacula   (to your top level directory)
cvs diff -u >my-changes.patch
\end{verbatim}

When the command is done, you can look in the file my-changes.patch
and you will see all the changes you have made to your copy of the
repository.  Make sure that you understand all the changes that
it reports before proceeding.  If you modified files that you do
do not want to commit to the main repository, you can simply delete
them from your local directory, and they will be restored from the
repository with the "cvs update" that is shown below.  Normally, you
should not find changes to files that you do not want to commit, and
if you find yourself in that position a lot, you are probably doing
something wrong.

Let's assume that now you want to commit your changes to the main CVS
repository.

First do:

\begin{verbatim}
cvs bacula
cvs update
\end{verbatim}

When you do this, it will pull any changes made by other developers into
your local copy of the repository, and it will check for conflicts. If there
are any, it will tell you, and you will need to resolve them.  The problems
of resolving conflicts are a bit more than this document can cover, but
you can examine the files it claims have conflicts and look for \lt{}\lt{}\lt{}\lt{}
or look in the .rej files that it creates.  If you have problems, just ask
on the developer's list.

Note, doing the above "cvs update" is not absolutely necessary.  There are
times when you may be working on code and you want to commit it, but you
explicitly do not want to move up to the latest version of the code in
the CVS.  If that is the case, you can simply skip the "cvs update" and
do the commit shown below.  If the commit fails because of a conflict, it
will tell you, and you must resolve the conflict before it will permit
you to do the commit.

Once your local copy of the repository has been updated, you can now
commit your changes:

\begin{verbatim}
cvs commit -m "Some comment about what you changed"
\end{verbatim}

or if you really only want to commit a single file, you can
do:

\begin{verbatim}
cvs commit -m "comment" scripts/file-I-edited
\end{verbatim}

Note, if you have done a build in your directory, or you have added
other new files, the commit will update only the files that are
actually in the repository.  For example, none of the object files
are stored in the repository, so when you do a commit, those object
files will simply be ignored.

If you want to add new files or remove files from the main CVS
repository, and you are not experienced with CVS, please ask Kern
to do it.  If you follow the simple steps above, it is unlikely that
you will do any damage to the repository, and if you do, it is always
possible for us to recover, but it can be painful.

If you are only working in one subdirectory of say the bacula project,
for example, the scripts directory, you can do your commit from
that subdirectory, and only the changes in that directory and all its
subdirectories will be committed.  This can be helpful for translators.
If you are doing a French translation, you will be working in
docs/manual-fr, and if you are always cd'ed into that directory when
doing your commits, your commit will effect only that directory.  As
long as you are careful only to change files that you want changed,
you have little to worry about.

\subsection*{The Development Cycle}
\index{Developement Cycle}
\index{Cycle!Developement}
\addcontentsline{toc}{subsubsection}{Development Cycle}

As I noted in the 1.38 ReleaseNotes, version 1.38 was different from prior
versions because it had a lot more contributions.  I expect that this trend
will continue.  As a consequence, I am going to modify how I normally do
development, and instead of making a list of all the features that I will
implement in the next version, I will personally sign up for one (maybe
two) projects at a time, and when they are complete, I will release a new
version.

The difference is that I will have more time to review the new code that is
being contributed, and will be able to devote more time to a smaller number
of projects (1.38 had too many new features for me to handle correctly).

I expect that future release schedules will be much the same, and the
number of new features will also be much the same providing that the
contributions continue to come -- and they show no signs of let up :-)

\index{Feature Requests}
{\bf Feature Requests:} \\
In addition, I would like to "formalize" the feature requests a bit.

Instead of me maintaining an informal list of everything I run into 
(kernstodo), I would like to maintain a "formal" list of projects.  This 
means that all new feature requests, including those recently discussed on 
the email lists, must be formally submitted and approved. 

Formal submission of feature requests will take two forms: \\
1. non-mandatory, but highly recommended is to discuss proposed new features
on the mailing list.\\
2.  Formal submission of an Feature Request in a special format.
I'll give an example of this below, but you can also find it on the web
site under "Support -\gt{} Feature Requests".  Since it takes a bit of time to
properly fill out a Feature Request form, you probably should check on the email list
first.

Once I receive the Feature Request, I will either accept it, send it back
asking for clarification, send it to the email list asking for opinions, or
reject it.

If it is accepted, it will go in the "projects" file (a simple ASCII file) 
maintained in the main Bacula source directory.

{\bf Implementation of Feature Requests:}\\
Any qualified developer can sign up for a project.  The project must have
an entry in the projects file, and the developer's name will appear in the
Status field.

{\bf How Feature Requests are accepted:}\\
Acceptance of Feature Requests depends on several things: \\
1.  feedback from users.  If it is negative, the Feature Request will probably not be
accepted.  \\
2.  the difficulty of the project.  A project that is so
difficult that I cannot imagine finding someone to implement probably won't
be accepted. \\
 3.  whether or not the Feature Request fits within the
current stategy of Bacula (for example an Feature Request that requests changing the
tape to tar format would not be accepted, ...)

{\bf How Feature Requests are prioritized:}\\
Once an Feature Request is accepted, it needs to be implemented.  If you
can find a developer for it, or one signs up for implementing it, then the
Feature Request becomes top priority (at least for that developer).

Between releases of Bacula, I will generally solicit Feature Request input
for the next version, and by way of this email, I suggest that you send
discuss and send in your Feature Requests for the next release.  Please
verify that the Feature Request is not in the current list (attached to this email).

Once users have had several weeks to submit Feature Requests, I will
organize them, and request users to vote on them.  This will allow fixing
prioritizing the Feature Requests.  Having a priority is one thing, but
getting it implement is another thing -- I am hoping that the Bacula
community will take more responsibility for assuring the implementation of
accepted Feature Requests.

Feature Request format:
\begin{verbatim}
============= Empty Feature Request form ===========
Item n:   One line summary ...
  Date:   Date submitted
  Origin: Name and email of originator.
  Status:

  What:   More detailed explanation ...

  Why:    Why it is important ...

  Notes:  Additional notes or features (omit if not used)
============== End Feature Request form ==============
\end{verbatim}

\begin{verbatim}
============= Example Completed  Feature Request form ===========
Item 1:   Implement a Migration job type that will move the job
          data from one device to another.
  Origin: Sponsored by Riege Sofware International GmbH. Contact:
          Daniel Holtkamp <holtkamp at riege dot com>
  Date:   28 October 2005
  Status: Partially coded in 1.37 -- much more to do. Assigned to
          Kern.

  What:   The ability to copy, move, or archive data that is on a
          device to another device is very important.

  Why:    An ISP might want to backup to disk, but after 30 days
          migrate the data to tape backup and delete it from
          disk.  Bacula should be able to handle this
          automatically.  It needs to know what was put where,
          and when, and what to migrate -- it is a bit like
          retention periods.  Doing so would allow space to be
          freed up for current backups while maintaining older
          data on tape drives.

  Notes:  Migration could be triggered by:
           Number of Jobs
           Number of Volumes
           Age of Jobs
           Highwater size (keep total size)
           Lowwater mark
=================================================
\end{verbatim}


\subsection*{Developing Bacula}
\index{Developing Bacula}
\index{Bacula!Developing}
\addcontentsline{toc}{subsubsection}{Developing Bacula}

Typically the simplest way to develop Bacula is to open one xterm window
pointing to the source directory you wish to update; a second xterm window at
the top source directory level, and a third xterm window at the bacula
directory \lt{}top\gt{}/src/bacula. After making source changes in one of the
directories, in the top source directory xterm, build the source, and start
the daemons by entering: 

make and 

./startit then in the enter: 

./console or 

./gnome-console to start the Console program. Enter any commands for testing.
For example: run kernsverify full. 

Note, the instructions here to use {\bf ./startit} are different from using a
production system where the administrator starts Bacula by entering {\bf
./bacula start}. This difference allows a development version of {\bf Bacula}
to be run on a computer at the same time that a production system is running.
The {\bf ./startit} strip starts {\bf Bacula} using a different set of
configuration files, and thus permits avoiding conflicts with any production
system. 

To make additional source changes, exit from the Console program, and in the
top source directory, stop the daemons by entering: 

./stopit then repeat the process. 

\subsubsection*{Debugging}
\index{Debugging}
\addcontentsline{toc}{subsubsection}{Debugging}

Probably the first thing to do is to turn on debug output. 

A good place to start is with a debug level of 20 as in {\bf ./startit -d20}.
The startit command starts all the daemons with the same debug level.
Alternatively, you can start the appropriate daemon with the debug level you
want. If you really need more info, a debug level of 60 is not bad, and for
just about everything a level of 200. 

\subsubsection*{Using a Debugger}
\index{Using a Debugger}
\index{Debugger!Using a}
\addcontentsline{toc}{subsubsection}{Using a Debugger}

If you have a serious problem such as a segmentation fault, it can usually be
found quickly using a good multiple thread debugger such as {\bf gdb}. For
example, suppose you get a segmentation violation in {\bf bacula-dir}. You
might use the following to find the problem: 

\lt{}start the Storage and File daemons\gt{}
cd dird
gdb ./bacula-dir
run -f -s -c ./dird.conf
\lt{}it dies with a segmentation fault\gt{}
where
The {\bf -f} option is specified on the {\bf run} command to inhibit {\bf
dird} from going into the background. You may also want to add the {\bf -s}
option to the run command to disable signals which can potentially interfere
with the debugging. 

As an alternative to using the debugger, each {\bf Bacula} daemon has a built
in back trace feature when a serious error is encountered. It calls the
debugger on itself, produces a back trace, and emails the report to the
developer. For more details on this, please see the chapter in the main Bacula
manual entitled ``What To Do When Bacula Crashes (Kaboom)''. 

\subsubsection*{Memory Leaks}
\index{Leaks!Memory}
\index{Memory Leaks}
\addcontentsline{toc}{subsubsection}{Memory Leaks}

Because Bacula runs routinely and unattended on client and server machines, it
may run for a long time. As a consequence, from the very beginning, Bacula
uses SmartAlloc to ensure that there are no memory leaks. To make detection of
memory leaks effective, all Bacula code that dynamically allocates memory MUST
have a way to release it. In general when the memory is no longer needed, it
should be immediately released, but in some cases, the memory will be held
during the entire time that Bacula is executing. In that case, there MUST be a
routine that can be called at termination time that releases the memory. In
this way, we will be able to detect memory leaks. Be sure to immediately
correct any and all memory leaks that are printed at the termination of the
daemons. 

\subsubsection*{Special Files}
\index{Files!Special}
\index{Special Files}
\addcontentsline{toc}{subsubsection}{Special Files}

Kern uses files named 1, 2, ... 9 with any extension as scratch files. Thus
any files with these names are subject to being rudely deleted at any time. 

\subsubsection*{When Implementing Incomplete Code}
\index{Code!When Implementing Incomplete}
\index{When Implementing Incomplete Code}
\addcontentsline{toc}{subsubsection}{When Implementing Incomplete Code}

Please identify all incomplete code with a comment that contains 

\begin{verbatim}
***FIXME***
\end{verbatim} 

where there are three asterisks (*) before and after the word
FIXME (in capitals) and no intervening spaces. This is important as it allows
new programmers to easily recognize where things are partially implemented. 

\subsubsection*{Bacula Source File Structure}
\index{Structure!Bacula Source File}
\index{Bacula Source File Structure}
\addcontentsline{toc}{subsubsection}{Bacula Source File Structure}

The distribution generally comes as a tar file of the form {\bf
bacula.x.y.z.tar.gz} where x, y, and z are the version, release, and update
numbers respectively. 

Once you detar this file, you will have a directory structure as follows: 

\footnotesize
\begin{verbatim}
|
Tar file:
|- depkgs
   |- mtx              (autochanger control program + tape drive info)
   |- sqlite           (SQLite database program)

Tar file:
|- depkgs-win32
   |- pthreads         (Native win32 pthreads library -- dll)
   |- zlib             (Native win32 zlib library)
   |- wx               (wxWidgets source code)

Project bacula:
|- bacula              (main source directory containing configuration
   |                    and installation files)
   |- autoconf         (automatic configuration files, not normally used
   |                    by users)
   |- intl             (programs used to translate)
   |- platforms        (OS specific installation files)
      |- redhat        (Red Hat installation)
      |- solaris       (Sun installation)
      |- freebsd       (FreeBSD installation)
      |- irix          (Irix installation -- not tested)
      |- unknown       (Default if system not identified)
   |- po               (translations of source strings)
   |- src              (source directory; contains global header files)
      |- cats          (SQL catalog database interface directory)
      |- console       (bacula user agent directory)
      |- dird          (Director daemon)
      |- filed         (Unix File daemon)
         |- win32      (Win32 files to make bacula-fd be a service)
      |- findlib       (Unix file find library for File daemon)
      |- gnome-console (GNOME version of console program)
      |- lib           (General Bacula library)
      |- stored        (Storage daemon)
      |- tconsole      (Tcl/tk console program -- not yet working)
      |- testprogs     (test programs -- normally only in Kern's tree)
      |- tools         (Various tool programs)
      |- win32         (Native Win32 File daemon)
         |- baculafd   (Visual Studio project file)
         |- compat     (compatibility interface library)
         |- filed      (links to src/filed)
         |- findlib    (links to src/findlib)
         |- lib        (links to src/lib)
         |- console    (beginning of native console program)
         |- wx-console (wxWidget console Win32 specific parts)
     |- wx-console     (wxWidgets console main source program)

Project regress:
|- regress             (Regression scripts)
   |- bin              (temporary directory to hold Bacula installed binaries)
   |- build            (temporary directory to hold Bacula source)
   |- scripts          (scripts and .conf files)
   |- tests            (test scripts)
   |- tmp              (temporary directory for temp files)
   |- working          (temporary working directory for Bacula daemons)

Project docs:
|- docs                (documentation directory)
   |- developers       (Developer's guide)
   |- home-page        (Bacula's home page source)
   |- manual           (html document directory)
   |- manual-fr        (French translation) 
   |- manual-de        (German translation) 
   |- techlogs         (Technical development notes);

Project rescue:
|- rescue              (Bacula rescue CDROM)
   |- linux            (Linux rescue CDROM)
      |- cdrom         (Linux rescue CDROM code)
        ...
   |- solaris          (Solaris rescue -- incomplete) 
   |- freebsd          (FreeBSD rescue -- incomplete)

Project gui:
|- gui                 (Bacula GUI projects)
  |- bacula-web        (Bacula web php management code)
  |- bimagemgr         (Web application for burning CDROMs)


\end{verbatim}
\normalsize

\subsubsection*{Header Files}
\index{Header Files}
\index{Files!Header}
\addcontentsline{toc}{subsubsection}{Header Files}

Please carefully follow the scheme defined below as it permits in general only
two header file includes per C file, and thus vastly simplifies programming.
With a large complex project like Bacula, it isn't always easy to ensure that
the right headers are invoked in the right order (there are a few kludges to
make this happen -- i.e. in a few include files because of the chicken and egg
problem, certain references to typedefs had to be replaced with {\bf void} ). 

Every file should include {\bf bacula.h}. It pulls in just about everything,
with very few exceptions. If you have system dependent ifdefing, please do it
in {\bf baconfig.h}. The version number and date are kept in {\bf version.h}. 

Each of the subdirectories (console, cats, dird, filed, findlib, lib, stored,
...) contains a single directory dependent include file generally the name of
the directory, which should be included just after the include of {\bf
bacula.h}. This file (for example, for the dird directory, it is {\bf dird.h})
contains either definitions of things generally needed in this directory, or
it includes the appropriate header files. It always includes {\bf protos.h}.
See below. 

Each subdirectory contains a header file named {\bf protos.h}, which contains
the prototypes for subroutines exported by files in that directory. {\bf
protos.h} is always included by the main directory dependent include file. 

\subsubsection*{Programming Standards}
\index{Standards!Programming}
\index{Programming Standards}
\addcontentsline{toc}{subsubsection}{Programming Standards}

For the most part, all code should be written in C unless there is a burning
reason to use C++, and then only the simplest C++ constructs will be used.
Note, Bacula is slowly evolving to use more and more C++. 

Code should have some documentation -- not a lot, but enough so that I can
understand it. Look at the current code, and you will see that I document more
than most, but am definitely not a fanatic. 

I prefer simple linear code where possible. Gotos are strongly discouraged
except for handling an error to either bail out or to retry some code, and
such use of gotos can vastly simplify the program. 

Remember this is a C program that is migrating to a {\bf tiny} subset of C++,
so be conservative in your use of C++ features. 

\subsubsection*{Do Not Use}
\index{Use!Do Not}
\index{Do Not Use}
\addcontentsline{toc}{subsubsection}{Do Not Use}

\begin{itemize}
\item STL -- it is totally incomprehensible. 
   \end{itemize}

\subsubsection*{Avoid if Possible}
\index{Possible!Avoid if}
\index{Avoid if Possible}
\addcontentsline{toc}{subsubsection}{Avoid if Possible}

\begin{itemize}
\item Using {\bf void *} because this generally means that one must
   using casting, and in C++ casting is rather ugly.  It is OK to use
   void * to pass structure address where the structure is not known  
   to the routines accepting the packet (typically callback routines).
   However, declaring "void *buf" is a bad idea.  Please use the
   correct types whenever possible.
\item Using undefined storage specifications such as (short, int, long,
   long long, size_t ...).  The problem with all these is that the number of bytes
   they allocate depends on the compiler and the system.  Instead use
   Bacula's types (int8_t, uint8_t, int32_t, uint32_t, int64_t, and 
   uint64_t).  This guarantees that the variables are given exactly the
   size you want. Please try at all possible to avoid using size_t ssize_t
   and the such. They are very system dependent.  However, some system
   routines may need them, so their use is often unavoidable.
\item Returning a malloc'ed buffer from a subroutine --  someone will forget
   to release it. 
\item Using reference variables -- it allows subroutines to  create side
   effects. Reference variables are OK, if you are sure the variable 
   *always* exists, and you are sure you can handle the side effects of
   a subroutine changing the "pointer".
\item Heap allocation (malloc) unless needed -- it is expensive. 
\item Templates -- they can create portability problems. 
\item Fancy or tricky C or C++ code, unless you give a  good explanation of
   why you used it. 
\item Too much inheritance -- it can complicate the code,  and make reading it
   difficult (unless you are in love  with colons) 
   \end{itemize}

\subsubsection*{Do Use Whenever Possible}
\index{Possible!Do Use Whenever}
\index{Do Use Whenever Possible}
\addcontentsline{toc}{subsubsection}{Do Use Whenever Possible}

\begin{itemize}
\item Locking and unlocking within a single subroutine.  
\item Malloc and free within a single subroutine.  
\item Comments and global explanations on what your code or  algorithm does. 
   \end{itemize}

\subsubsection*{Indenting Standards}
\index{Standards!Indenting}
\index{Indenting Standards}
\addcontentsline{toc}{subsubsection}{Indenting Standards}

I cannot stand code indented 8 columns at a time. This makes the code
unreadable. Even 4 at a time uses a lot of space, so I have adopted indenting
3 spaces at every level. Note, indention is the visual appearance of the
source on the page, while tabbing is replacing a series of up to 8 spaces from
a tab character. 

The closest set of parameters for the Linux {\bf indent} program that will
produce reasonably indented code are: 

\footnotesize
\begin{verbatim}
-nbad -bap -bbo -nbc -br -brs -c36 -cd36 -ncdb -ce -ci3 -cli0
-cp36 -d0 -di1 -ndj -nfc1 -nfca -hnl -i3 -ip0 -l85 -lp -npcs
-nprs -npsl -saf -sai -saw -nsob -nss -nbc -ncs -nbfda
\end{verbatim}
\normalsize

You can put the above in your .indent.pro file, and then just invoke indent on
your file. However, be warned. This does not produce perfect indenting, and it
will mess up C++ class statements pretty badly. 

Braces are required in all if statements (missing in some very old code). To
avoid generating too many lines, the first brace appears on the first line
(e.g. of an if), and the closing brace is on a line by itself. E.g. 

\footnotesize
\begin{verbatim}
   if (abc) {
      some_code;
  }
\end{verbatim}
\normalsize

Just follow the convention in the code. Originally I indented case clauses
under a switch(), but now I prefer non-indented cases. 

\footnotesize
\begin{verbatim}
   switch (code) {
   case 'A':
      do something
      break;
   case 'B':
      again();
      break;
   default:
      break;
  }
\end{verbatim}
\normalsize

Avoid using // style comments except for temporary code or turning off debug
code. Standard C comments are preferred (this also keeps the code closer to
C). 

Attempt to keep all lines less than 85 characters long so that the whole line
of code is readable at one time. This is not a rigid requirement. 

Always put a brief description at the top of any new file created describing
what it does and including your name and the date it was first written. Please
don't forget any Copyrights and acknowledgments if it isn't 100\% your code.
Also, include the Bacula copyright notice that is in {\bf src/c}. 

In general you should have two includes at the top of the an include for the
particular directory the code is in, for includes are needed, but this should
be rare. 

In general (except for self-contained packages), prototypes should all be put
in {\bf protos.h} in each directory. 

Always put space around assignment and comparison operators. 

\footnotesize
\begin{verbatim}
   a = 1;
   if (b >= 2) {
     cleanup();
  }
\end{verbatim}
\normalsize

but your can compress things in a {\bf for} statement: 

\footnotesize
\begin{verbatim}
   for (i=0; i < del.num_ids; i++) {
    ...
\end{verbatim}
\normalsize

Don't overuse the inline if (?:). A full {\bf if} is preferred, except in a
print statement, e.g.: 

\footnotesize
\begin{verbatim}
   if (ua->verbose \&& del.num_del != 0) {
      bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del,
         del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName);
  }
\end{verbatim}
\normalsize

Leave a certain amount of debug code (Dmsg) in code you submit, so that future
problems can be identified. This is particularly true for complicated code
likely to break. However, try to keep the debug code to a minimum to avoid
bloating the program and above all to keep the code readable. 

Please keep the same style in all new code you develop. If you include code
previously written, you have the option of leaving it with the old indenting
or re-indenting it. If the old code is indented with 8 spaces, then please
re-indent it to Bacula standards. 

If you are using {\bf vim}, simply set your tabstop to 8 and your shiftwidth
to 3. 

\subsubsection*{Tabbing}
\index{Tabbing}
\addcontentsline{toc}{subsubsection}{Tabbing}

Tabbing (inserting the tab character in place of spaces) is as normal on all
Unix systems -- a tab is converted space up to the next column multiple of 8.
My editor converts strings of spaces to tabs automatically -- this results in
significant compression of the files. Thus, you can remove tabs by replacing
them with spaces if you wish. Please don't confuse tabbing (use of tab
characters) with indenting (visual alignment of the code). 

\subsubsection*{Don'ts}
\index{Don'ts}
\addcontentsline{toc}{subsubsection}{Don'ts}

Please don't use: 

\footnotesize
\begin{verbatim}
strcpy()
strcat()
strncpy()
strncat();
sprintf()
snprintf()
\end{verbatim}
\normalsize

They are system dependent and un-safe. These should be replaced by the Bacula
safe equivalents: 

\footnotesize
\begin{verbatim}
char *bstrncpy(char *dest, char *source, int dest_size);
char *bstrncat(char *dest, char *source, int dest_size);
int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...);
int bvsnprintf(char *str, int32_t size, const char  *format, va_list ap);
\end{verbatim}
\normalsize

See src/lib/bsys.c for more details on these routines. 

Don't use the {\bf \%lld} or the {\bf \%q} printf format editing types to edit
64 bit integers -- they are not portable. Instead, use {\bf \%s} with {\bf
edit\_uint64()}. For example: 

\footnotesize
\begin{verbatim}
   char buf[100];
   uint64_t num = something;
   char ed1[50];
   bsnprintf(buf, sizeof(buf), "Num=%s\n", edit_uint64(num, ed1));
\end{verbatim}
\normalsize

The edit buffer {\bf ed1} must be at least 27 bytes long to avoid overflow.
See src/lib/edit.c for more details. If you look at the code, don't start
screaming that I use {\bf lld}. I actually use subtle trick taught to me by
John Walker. The {\bf lld} that appears in the editing routine is actually
{\bf \#define} to a what is needed on your OS (usually ``lld'' or ``q'') and
is defined in autoconf/configure.in for each OS. C string concatenation causes
the appropriate string to be concatenated to the ``\%''. 

Also please don't use the STL or Templates or any complicated C++ code. 

\subsubsection*{Message Classes}
\index{Classes!Message}
\index{Message Classes}
\addcontentsline{toc}{subsubsection}{Message Classes}

Currently, there are five classes of messages: Debug, Error, Job, Memory, 
and Queued.

\subsubsection*{Debug Messages}
\index{Messages!Debug}
\index{Debug Messages}
\addcontentsline{toc}{subsubsection}{Debug Messages}

Debug messages are designed to be turned on at a specified debug level and are
always sent to STDOUT. There are designed to only be used in the development
debug process. They are coded as: 

DmsgN(level, message, arg1, ...) where the N is a number indicating how many
arguments are to be substituted into the message (i.e. it is a count of the
number arguments you have in your message -- generally the number of percent
signs (\%)). {\bf level} is the debug level at which you wish the message to
be printed. message is the debug message to be printed, and arg1, ... are the
arguments to be substituted. Since not all compilers support \#defines with
varargs, you must explicitly specify how many arguments you have. 

When the debug message is printed, it will automatically be prefixed by the
name of the daemon which is running, the filename where the Dmsg is, and the
line number within the file. 

Some actual examples are: 

Dmsg2(20, ``MD5len=\%d MD5=\%s\textbackslash{}n'', strlen(buf), buf); 

Dmsg1(9, ``Created client \%s record\textbackslash{}n'', client->hdr.name); 

\subsubsection*{Error Messages}
\index{Messages!Error}
\index{Error Messages}
\addcontentsline{toc}{subsubsection}{Error Messages}

Error messages are messages that are related to the daemon as a whole rather
than a particular job. For example, an out of memory condition my generate an
error message. They should be very rarely needed. In general, you should be
using Job and Job Queued messages (Jmsg and Qmsg). They are coded as: 

EmsgN(error-code, level, message, arg1, ...) As with debug messages, you must
explicitly code the of arguments to be substituted in the message. error-code
indicates the severity or class of error, and it may be one of the following: 

\addcontentsline{lot}{table}{Message Error Code Classes}
\begin{longtable}{lp{3in}}
{{\bf M\_ABORT} } & {Causes the daemon to immediately abort. This should be
used only  in extreme cases. It attempts to produce a  traceback. } \\
{{\bf M\_ERROR\_TERM} } & {Causes the daemon to immediately terminate. This
should be used only  in extreme cases. It does not produce a  traceback. } \\
{{\bf M\_FATAL} } & {Causes the daemon to terminate the current job, but the
daemon keeps running } \\
{{\bf M\_ERROR} } & {Reports the error. The daemon and the job continue
running } \\
{{\bf M\_WARNING} } & {Reports an warning message. The daemon and the job
continue running } \\
{{\bf M\_INFO} } & {Reports an informational message.}

\end{longtable}

There are other error message classes, but they are in a state of being
redesigned or deprecated, so please do not use them. Some actual examples are:


Emsg1(M\_ABORT, 0, ``Cannot create message thread: \%s\textbackslash{}n'',
strerror(status)); 

Emsg3(M\_WARNING, 0, ``Connect to File daemon \%s at \%s:\%d failed. Retrying
...\textbackslash{}n'',  client-\gt{}hdr.name, client-\gt{}address,
client-\gt{}port); 

Emsg3(M\_FATAL, 0, ``bdird\lt{}filed: bad response from Filed to \%s command:
\%d \%s\textbackslash{}n'',  cmd, n, strerror(errno)); 

\subsubsection*{Job Messages}
\index{Job Messages}
\index{Messages!Job}
\addcontentsline{toc}{subsubsection}{Job Messages}

Job messages are messages that pertain to a particular job such as a file that
could not be saved, or the number of files and bytes that were saved. They
Are coded as:
\begin{verbatim}
Jmsg(jcr, M\_FATAL, 0, "Text of message");
\end{verbatim}
A Jmsg with M\_FATAL will fail the job. The Jmsg() takes varargs so can
have any number of arguments for substituted in a printf like format.
Output from the Jmsg() will go to the Job report.
<br>
If the Jmsg is followed with a number such as Jmsg1(...), the number
indicates the number of arguments to be substituted (varargs is not
standard for \#defines), and what is more important is that the file and
line number will be prefixed to the message. This permits a sort of debug
from user's output.

\subsubsection*{Queued Job Messages}
\index{Queued Job Messages}
\index{Messages!Job}
\addcontentsline{toc}{subsubsection}{Queued Job Messages}
Queued Job messages are similar to Jmsg()s except that the message is
Queued rather than immediately dispatched. This is necessary within the
network subroutines and in the message editing routines. This is to prevent
recursive loops, and to ensure that messages can be delivered even in the
event of a network error.


\subsubsection*{Memory Messages}
\index{Messages!Memory}
\index{Memory Messages}
\addcontentsline{toc}{subsubsection}{Memory Messages}

Memory messages are messages that are edited into a memory buffer. Generally
they are used in low level routines such as the low level device file dev.c in
the Storage daemon or in the low level Catalog routines. These routines do not
generally have access to the Job Control Record and so they return error
messages reformatted in a memory buffer. Mmsg() is the way to do this.