ebl Add notes about UseStatistics

[bacula/docs] / docs / manuals / en / concepts / newfeatures.tex

%%
%%

\chapter{New Features}
\label{NewFeaturesChapter}
\index[general]{New Features}

This chapter presents the new features added to the development 2.5.x
versions to be released as Bacula version 3.0.0 near the end of 2008.

\section{Copy Jobs}
\index[general]{Copy Jobs}
A new {\bf Copy} job type has been implemented. It is essentially
identical to the existing Migration feature with the exception that
the Job that is copied is left unchanged.  This essentially creates
two identical copies of the same backup.  The Copy Job runs without
using the File daemon by copying the data from the old backup Volume to
a different Volume in a different Pool. See the Migration documentation
for additional details.

\section{Virtual Backup (Vbackup)}
\index[general]{Virtual Backup}
\index[general]{Vbackup}

Bacula's virtual backup feature is often called Synthetic Backup or
Consolidation in other backup products.  It permits you to consolidate
the previous Full backup plus the most recent Differential backup and any
subsequent Incremental backups into a new Full backup. This is accomplished
without contacting the client by reading the previous backup data and 
writing it to a volume in a different pool.  

In some respects the Vbackup feature works similar to a Migration job, in
that Bacula normally reads the data from the pool specified in the 
Job resource, and writes it to the {\bf Next Pool} specified in the 
Job resource.  The input Storage resource and the Output Storage resource
must be different.

The Vbackup is enabled on a Job by Job in the Job resource by specifying
a level of {\bf VirtualFull}.

A typical Job resource definition might look like the following:

\begin{verbatim}
Job {
  Name = "MyBackup"
  Type = Backup
  Client=localhost-fd
  FileSet = "Full Set"
  Storage = File
  Messages = Standard
  Pool = Default
  SpoolData = yes
}

# Default pool definition
Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365d             # one year
  NextPool = Full
  Storage = File
}

Pool {
  Name = Full
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365d             # one year
  Storage = DiskChanger
}

# Definition of file storage device
Storage {
  Name = File
  Address = localhost
  Password = "xxx"
  Device = FileStorage
  Media Type = File
  Maximum Concurrent Jobs = 5
}

# Definition of DDS Virtual tape disk storage device
Storage {
  Name = DiskChanger
  Address = localhost                # N.B. Use a fully qualified name here
  Password = "yyy"
  Device = DiskChanger
  Media Type = DiskChangerMedia
  Maximum Concurrent Jobs = 4
  Autochanger = yes
}
\end{verbatim}

Then in bconsole or via a Run schedule, you would run the job as:

\begin{verbatim}
run job=MyBackup level=Full
run job=MyBackup level=Incremental
run job=MyBackup level=Differential
run job=MyBackup level=Incremental
run job=MyBackup level=Incremental
\end{verbatim}

So providing there were changes between each of those jobs, you would end up
with a Full backup, a Differential, which includes the first Incremental
backup, then two Incremental backups.  All the above jobs would be written to
the {\bf Default} pool.

To consolidate those backups into a new Full backup, you would run the
following:

\begin{verbatim}
run job=MyBackup level=VirtualFull
\end{verbatim}

And it would produce a new Full backup without using the client, and the output
would be written to the {\bf Full} Pool which uses the Diskchanger Storage.

\section{TLS Authentication}
\index[general]{TLS Authentication}
In Bacula version 2.5.x and later, in addition to the normal Bacula
CRAM-MD5 authentication that is used to authenticate each Bacula
connection, you can specify that you want TLS Authentication as well,
which will provide more secure authentication.

This new feature uses Bacula's existing TLS code (normally used for
communications encryption) to do authentication.  To use it, you must
specify all the TLS directives normally used to enable communications
encryption (TLS Enable, TLS Verify Peer, TLS Certificate, ...) and
a new directive:

\begin{verbatim}
TLS Authenticate = yes
\end{verbatim}

in the main daemon configuration resource (Director for the Director,
Client for the File daemon, and Storage for the Storage daemon).

When {\bf TLS Authenticate} is enabled, after doing the CRAM-MD5
authentication, Bacula will do the normal TLS authentication, then TLS 
encryption will be turned off.

If you want to encrypt communications data, do not turn on {\bf TLS
Authenticate}.

\section{bextract non-portable Win32 data}
\index[general]{bextract handles Win32 non-portable data}
{\bf bextract} has been enhanced to be able to restore
non-portable Win32 data to any OS.  Previous versions were 
unable to restore non-portable Win32 data to machines that
did not have the Win32 BackupRead and BackupWrite API calls.

\section{State File updated at Job Termination}
\index[general]{State File}
In previous versions of Bacula, the state file, which provides a
summary of previous jobs run in the {\bf status} command output was
updated only when Bacula terminated, thus if the daemon crashed, the
state file might not contain all the run data.  This version of
the Bacula daemons updates the state file on each job termination.

\section{MaxFullInterval}
\index[general]{MaxFullInterval}
The new Job resource directive {\bf Max Full Interval = \lt{}time-interval\gt{}}
can be used to specify the maximum time interval between {\bf Full} backup
jobs. When a job starts, if the time since the last Full backup is
greater than the specified interval, and the job would normally be an
{\bf Incremental} or {\bf Differential}, it will be automatically
upgraded to a {\bf Full} backup.

\section{MaxDiffInterval}
\index[general]{MaxDiffInterval}
The new Job resource directive {\bf Max Diff Interval = \lt{}time-interval\gt{}}
can be used to specify the maximum time interval between {\bf Differential} backup
jobs. When a job starts, if the time since the last Differential backup is
greater than the specified interval, and the job would normally be an
{\bf Incremental}, it will be automatically
upgraded to a {\bf Differential} backup.

\section{No Dump Flag}
\index[general]{MaxDiffInterval}
On FreeBSD systems, each file has a {\bf no dump flag} that can be set
by the user, and when it is set it is an indication to backup programs
to not backup that particular file.  This version of Bacula contains a
new Options directive within a FileSet resource, which instructs Bacula to
obey this flag.  The new directive is:

\begin{verbatim}
  Honor No Dump Flag = yes|no
\end{verbatim}

The default value is {\bf no}.


\section{Duplicate Job Control}
\index[general]{Duplicate Jobs}
The new version of Bacula provides four new directives that
give additional control over what Bacula does if duplicate jobs 
are started.  A duplicate job in the sense we use it here means
a second or subsequent job with the same name starts.  This
happens most frequently when the first job runs longer than expected because no 
tapes are available.

The four directives each take as an argument a yes or no value and
are specified in the Job resource.

They are:

\begin{description}
\item [Allow Duplicate Jobs = \lt{}yes|no\gt{}]
  If this directive is enabled duplicate jobs will be run.  If
  the directive is set to {\bf no} (default) then only one job of a given name
  may run at one time, and the action that Bacula takes to ensure only
  one job runs is determined by the other directives (see below).

\item [Allow Higher Duplicates = \lt{}yes|no\gt{}]
  If this directive is set to {\bf yes} (default) the job with a higher
  priority (lower priority number) will be permitted to run.  If the
  priorities of the two jobs are the same, the outcome is determined by
  other directives (see below).

\item [Cancel Queued Duplicates = \lt{}yes|no\gt{}]
  If this directive is set to {\bf yes} (default) any job that is
  already queued to run but not yet running will be canceled.

\item [Cancel Running Duplicates = \lt{}yes|no\gt{}]
  If this directive is set to {\bf yes} any job that is already running
  will be canceled.  The default is {\bf no}.
\end{description}


\section{Ignore Dir}
\index[general]{IgnoreDir}
The {\bf Ignore Dir = \lt{}filename\gt{}} is a new directive that can be added to the Include
section of the FileSet resource.  If the specified
filename is found on the Client in any directory to be backed up, 
the whole directory will be ignored (not backed up).
For example:

\begin{verbatim}
    # List of files to be backed up
   FileSet {
       Name = "MyFileSet"
       Include {
           Options {
               signature = MD5
           }
           File = /home
           IgnoreDir = .excludeme
       }
   }
\end{verbatim}

But in /home, there may be hundreds of directories of users and some
people want to indicate that they don't want to have certain
directories backed up. For example, with the above FileSet, if
the user or sysadmin creates a file named {\bf .excludeme} in 
specific directories, such as

\begin{verbatim}
   /home/user/www/cache/.excludeme
   /home/user/temp/.excludeme
\end{verbatim}

then Bacula will not backup the two directories named:

\begin{verbatim}
   /home/user/www/cache
   /home/user/temp
\end{verbatim}



\section{Accurate}
\index[general]{Accurate Backup}
As with most other backup programs, Bacula decides what files to backup 
for Incremental and Differental backup by comparing the change (st\_ctime)
and modification (st\_mtime) times of the file to the time the last backup completed.
If one of those two times is later than the last backup time, then the file
will be backed up.  This does not, however, permit tracking what files have
been deleted and will miss any file with an old time that may have been
restored or moved on the client filesystem.

If the {\bf Accurate = \lt{}yes|no\gt{}} directive is enabled (default no) in the
Job resource, the job will be run as an Accurate Job. For a {\bf Full}
backup, there is no difference, but for {\bf Differential} and {\bf Incremental}
backups, the Director will send a list of all previous files backed up, and the 
File daemon will use that list to determine if any new files have been added or
or moved and if any files have been deleted. This allows Bacula to make an accurate
backup of your system to that point in time so that if you do a restore, it
will restore your system exactly.  The downside of using Accurate backup is that
it requires significantly more resources (CPU and memory) on both the Director and
the Client machine to create the list of previous files backed up, to send that
list to the File daemon, and do comparisons on the File daemon between every file
and the list.

\section{Bacula Plugins}
\index[general]{Plugin}
Support for shared object plugins has been implemented in the Linux
(and Unix) File daemon. The API will be documented separately in
the Developer's Guide or in a new document.  For the moment, there is
a single plugin named {\bf bpipe} that allows an external program to
get control to backup and restore a file.

Plugins are also planned (partially implemented) in the Director and the
Storage daemon.  Also we plan (at some point) to port (partially implemented)
the plugin code to Win32 machines.


\section{Display Autochanger Content}
\index[general]{StatusSlots}

The {\bf status slots storage=\lt{}storage-name\gt{}} command displays autochanger content.

\footnotesize
\begin{verbatim}
 Slot |  Volume Name    |  Status  |  Media Type       |    Pool        |  Loaded |
------+-----------------+----------+-------------------+----------------+---------|
    1 |           00001 |   Append |  DiskChangerMedia |        Default |    0    |
    2 |           00002 |   Append |  DiskChangerMedia |        Default |    0    |
    3*|           00003 |   Append |  DiskChangerMedia |        Scratch |    0    |
    4 |                 |          |                   |                |    0    |
\end{verbatim}
\normalsize

If you an asterisk ({\bf *}) appears after the slot number, you must run an {\bf update slots}
command to synchronize autochanger content with your catalog.

\section{Miscellaneous}
\index[general]{Misc New Features}
\begin{description}
\item [Virtual Tape Emulation]
We now have a Virtual Tape emulator that allows us to run though 99.9\% of the 
tape code but actually reading and writing to a disk file. This feature is most
useful in testing.
It is enabled by using {\bf Device Type = vtape} in the Storage daemon's Device
directive. This feature is only implemented on Linux machines.

\item [Bat Enhancements]
Bat (the Bacula Administration Tool) GUI program has been significantly
enhanced and stabilized. In particular, there are new table based status 
commands; it can now be easily localized using Qt4 Linguist.

The Bat communications protocol has been significantly enhanced to improve
GUI handling.

\item [RunScript Enhancements]
The {\bf RunScript} resource has been enhanced to permit multiple
commands per RunScript.  Simply specify multiple {\bf Command} directives
in your RunScript.

A new Client RunScript {\bf RunsWhen} keyword of {\bf AfterVSS} has been implemented, which
runs the command after the Volume Shadow Copy has been made.

Console commands can be specified within a RunScript by using:
{\bf Console = \lt{}command\gt{}}, however, this command has not been 
carefully tested and debugged and is known to easily crash the Director.
We would appreciate feedback.  Due to the recursive nature of this command, we
may remove it before the final release.

\item [Status Enhancements]
The bconsole {\bf status dir} output has been enhanced to indicate
Storage daemon job spooling and despooling activity.

\item [Statistics Enhancements]

If you (or you boss) want to have statistics on your backups, you could use
some SQL stuffs on the Job table to report how many:
\begin{itemize}
\item jobs have run
\item jobs have been successful
\item files have been backuped
\item ...
\end{itemize}

Theses statistics are accurate only if your job retention is greater than
your statistic period. Ie, if jobs are purged from the catalog, you won't be
able to use them.

Now, you can configure bacula to copy Job records to a new JobStat table
with the \textbf{Use Statistics = \lt{}yes|no\gt{}} Job directive.

The \texttt{Statistics Retention = \lt{}time\gt{}} director directive defines
the length of time that Bacula will keep statistics job records in the Catalog
database after the Job End time. (In \texttt{JobStat} table) When this time
period expires, and if user runs \texttt{prune stats} command, Bacula will
prune (remove) Job records that are older than the specified period.

Theses statistics records aren't use for restore purpose, but mainly for
capacity planning, billings, etc.

\begin{verbatim}
Job {
  Name = ImportantBackup
  Client = rufus-fd
  ...
  Use Statistics = yes
}
\end{verbatim}


\end{description}