Add plugin documentation

[bacula/docs] / docs / manuals / en / main / newbsfeatures.tex

\chapter{New Features in Bacula Enterprise 6.0.x}
This chapter presents the new features that have been added to the
current Enterprise version of Bacula.
These features are available only with a Bacula Systems subscription.

In addition to the features in this chapter, the Enterprise version
will include the Community features described in the Community new Features
chapter.

\chapter{New Features in Bacula Enterprise 6.0.6}

\section{Incremental Accelerator Plugin for NetApp}

The Incremental Accelerator for NetApp Plugin is designed to simplify the
backup and restore procedure of your NetApp NAS hosting a huge number of files.

\smallskip{} When using the NetApp HFC Plugin, Bacula Enterprise will query the
NetApp device to get the list of all files modified since the last backup
instead of having to walk through the entire filesystem. Once Bacula have the
list of all files to back's up, it will use a standard network share (such as
NFS or CIFS) to access files.

\smallskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.

\section{PostgreSQL Plugin}

The PostgreSQL plugin is designed to simplify the backup and restore procedure
of your PostgreSQL cluster, the backup administrator doesn't need to learn about
internals of Postgres backup techniques or write complex scripts. The plugin
will automatically take care for you to backup essential information such as
configuration, users definition or tablespaces. The PostgreSQL plugin supports
both dump and Point In Time Recovery (PITR) backup techniques.

\smallskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.

\chapter{New Features in Bacula Enterprise 6.0.5}

\section{Maximum Reload Requests}

The new Director directive \texttt{Maximum Reload Requests} permits to
configure the number of reload requests that can be done while jobs are
running.

\begin{verbatim}
Director {
  Name = localhost-dir
  Maximum Reload Requests = 64
  ...

}
\end{verbatim}

\section{FD Storage Address}

When the Director is behind a NAT, in a WAN area, to connect to 
% the FileDaemon or 
the StorageDaemon, the Director uses an "external" ip address,
and the FileDaemon should use an "internal" ip address to contact the
StorageDaemon.

The normal way to handle this situation is to use a canonical name such as
"storage-server" that will be resolved on the Director side as the WAN address
and on the Client side as the LAN address. This is now possible to configure
this parameter using the new \texttt{FDStorageAddress} Storage 
% or Client
directive.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=10cm]{\idir BackupOverWan1}
  \label{fig:fdstorageaddress}
  \caption{Backup over WAN}
\end{figure}

\begin{verbatim}
Storage {
     Name = storage1
     Address = 65.1.1.1
     FD Storage Address = 10.0.0.1
     SD Port 9103
     ...
}
\end{verbatim}

% # or in the Client resouce
% 
% Client {
%      Name = client1
%      Address = 65.1.1.2
%      FD Storage Address = 10.0.0.1
%      FD Port 9103
%      ...
% }
% \end{verbatim}
% 
% Note that using the Client \texttt{FDStorageAddress} directive will not allow
% to use multiple Storage Daemon, all Backup or Restore requests will be sent to
% the specified \texttt{FDStorageAddress}.

\chapter{New Features in Bacula Enterprise 6.0.4}

\section{VMWare vSphere VADP Plugin}

The \bsee{} vSphere plugin provides virtual
machine bare metal recovery, while the backup at the guest level simplify data
protection of critical applications.

The plugin integrates the VMware's Changed Block Tracking (CBT) technology to
ensure only blocks that have changed since the initial Full, and/or the last
Incremental or Differential Backup are sent to the current Incremental or
Differential backup stream to give you more efficient backups and reduced
network load.

\section{Oracle RMAN Plugin}

The Bacula Enterprise Oracle Plugin is designed to simplify the backup and
restore procedure of your Oracle Database instance, the backup administrator
don't need to learn about internals of Oracle backup techniques or write
complex scripts.  The Bacula Enterprise Oracle plugin supports both dump and
Point In Time Recovery (PITR) with RMAN backup techniques.

\chapter{New Features in Bacula Enterprise 6.0.0}

\section{Incomplete Jobs}
During a backup, if the Storage daemon experiences disconnection
with the File daemon during backup (normally a comm line problem
or possibly an FD failure), under conditions that the SD determines
to be safe it will make the failed job as Incomplete rather than
failed.  This is done only if there is sufficient valid backup
data that was written to the Volume. The advantage of an Incomplete
job is that it can be restarted by the new bconsole {\bf restart}
command from the point where it left off rather than from the
beginning of the jobs as is the case with a cancel.

\section{The Stop Command}
Bacula has been enhanced to provide a {\bf stop} command,
very similar to the {\bf cancel} command with the main difference
that the Job that is stopped is marked as Incomplete so that
it can be restarted later by the {\bf restart} command where
it left off (see below).  The {\bf stop} command with no
arguments, will like the cancel command, prompt you with the
list of running jobs allowing you to select one, which might
look like the following:

\begin{verbatim}
*stop
Select Job:
     1: JobId=3 Job=Incremental.2012-03-26_12.04.26_07
     2: JobId=4 Job=Incremental.2012-03-26_12.04.30_08
     3: JobId=5 Job=Incremental.2012-03-26_12.04.36_09
Choose Job to stop (1-3): 2
2001 Job "Incremental.2012-03-26_12.04.30_08" marked to be stopped.
3000 JobId=4 Job="Incremental.2012-03-26_12.04.30_08" marked to be stopped.
\end{verbatim}

\section{The Restart Command}
The new {\bf Restart command} allows console users to restart
a canceled, failed, or incomplete Job.  For canceled and failed
Jobs, the Job will restart from the beginning.  For incomplete 
Jobs the Job will restart at the point that it was stopped either
by a stop command or by some recoverable failure.

\smallskip
If you enter the {\bf restart} command in bconsole, you will get the
following prompts:

\begin{verbatim}
*restart
You have the following choices:
     1: Incomplete
     2: Canceled
     3: Failed
     4: All
Select termination code:  (1-4): 
\end{verbatim}

If you select the {\bf All} option, you may see something like:

\begin{verbatim}
Select termination code:  (1-4): 4
+-------+-------------+---------------------+------+-------+----------+-----------+-----------+
| jobid | name        | starttime           | type | level | jobfiles |
jobbytes  | jobstatus |
+-------+-------------+---------------------+------+-------+----------+-----------+-----------+
|     1 | Incremental | 2012-03-26 12:15:21 | B    | F     |        0 |
    0 | A         |
|     2 | Incremental | 2012-03-26 12:18:14 | B    | F     |      350 |
4,013,397 | I         |
|     3 | Incremental | 2012-03-26 12:18:30 | B    | F     |        0 |
    0 | A         |
|     4 | Incremental | 2012-03-26 12:18:38 | B    | F     |      331 |
3,548,058 | I         |
+-------+-------------+---------------------+------+-------+----------+-----------+-----------+
Enter the JobId list to select: 
\end{verbatim}

Then you may enter one or more JobIds to be restarted, which may 
take the form of a list of JobIds separated by commas, and/or JobId
ranges such as {\bf 1-4}, which indicates you want to restart JobIds
1 through 4, inclusive.

\section{Support for Exchange Incremental Backups}
The Bacula Enterprise version 6.0 VSS plugin now supports
Full and Incremental backups for Exchange.  We strongly
recommend that you do not attempt to run Differential jobs with
Exchange as it is likely to produce a situation where restores
will no longer select the correct jobs, and thus the
Windows Exchange VSS writer will fail when applying log files.
There is a Bacula Systems Enterprise white paper that provides
the details of backup and restore of Exchange 2010 with the
Bacula VSS plugin.

\smallskip
Restores can be done while Exchange is running, but you
must first unmount (dismount in Microsoft terms) any database
you wish to restore and explicitly mark them to permit a
restore operation (see the white paper for details).

\smallskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.

\section{Support for MSSQL Block Level Backups}
With the addition of block level backup support to the 
Bacula Enterprise VSS MSSQL component, you can now do
Differential backups in addition to Full backups.
Differential backups use Microsoft's partial block backup
(a block differencing or deduplication that we call Delta).
This partial block backup permits backing up only those
blocks that have changed.  Database restores can be made while
the MSSQL server is running, but any databases selected for
restore will be automatically taken offline by the MSSQL 
server during the restore process.

Incremental backups for MSSQL are not support by
Microsoft. We strongly recommend that you not perform Incremental
backups with MSSQL as they will probably produce a situation
where restore will no longer work correctly. 

\smallskip
 We are currently working on producing a white paper that will give more
details of backup and restore with MSSQL. One point to note is that during
a restore, you will normally not want to restore the {\bf master} database.
You must exclude it from the backup selections that you have made or the
restore will fail.

\smallskip
It is possible to restore the {\bf master} database, but you must
first shutdown the MSSQL server, then you must perform special 
recovery commands.  Please see Microsoft documentation on how
to restore the master database.

\smallskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.


\section{Job Bandwidth Limitation}

The new {\bf Job Bandwidth Limitation} directive may be added to the File
daemon's and/or Director's configuration to limit the bandwidth used by a Job
on a Client.  It can be set in the File daemon's conf file for all Jobs run in
that File daemon, or it can be set for each Job in the Director's conf file.

For example:
\begin{verbatim}
FileDaemon {
  Name = localhost-fd
  Working Directory = /some/path
  Pid Directory = /some/path
  ...
  Maximum Bandwidth Per Job = 5Mb/s
}
\end{verbatim}

The above example would cause any jobs running with the FileDaemon to not
exceed 5Mb/s of throughput when sending data to the Storage Daemon.

You can specify the speed parameter in k/s, Kb/s, m/s, Mb/s.

For example:
\begin{verbatim}
Job {
  Name = locahost-data
  FileSet = FS_localhost
  Accurate = yes
  ...
  Maximum Bandwidth = 5Mb/s
  ...
}
\end{verbatim}

The above example would cause Job \texttt{localhost-data} to not exceed 5MB/s
of throughput when sending data from the File daemon to the Storage daemon.

A new console command \texttt{setbandwidth} permits to set dynamically the
maximum throughput of a running Job or for future jobs of a Client.

\begin{verbatim}
* setbandwidth limit=1000000 jobid=10
\end{verbatim}

The \texttt{limit} parameter is in Kb/s.

\medskip
This project was funded by Bacula Systems and is available in
the Enterprise Edition.

\section{Incremental/Differential Block Level Difference Backup}

The new \texttt{delta} Plugin is able to compute and apply signature-based file
differences. It can be used to backup only changes in a big binary file like
Outlook PST, VirtualBox/VMware images or database files.

It supports both Incremental and Differential backups and stores signatures
database in the File Daemon working directory. This plugin is available on all
platform including Windows 32 and 64bit.

Accurate option should be turned on in the Job resource.
\begin{verbatim}
Job {
 Accurate = yes
 FileSet = DeltaFS
 ...
}

FileSet {
 Name = DeltaFS
 ...
 Include {
   # Specify one file
   Plugin = "delta:/home/eric/.VirtualBox/HardDisks/lenny-i386.vdi"
 }
}

FileSet {
 Name = DeltaFS-Include
 ...
 Include {
   Options {
      Compression = GZIP1
      Signature = MD5
      Plugin = delta
   }
   # Use the Options{} filtering and options
   File = /home/user/.VirtualBox
 }
}

\end{verbatim}

Please contact Bacula Systems support to get Delta Plugin specific
documentation.

\medskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.

\section{SAN Shared Tape Storage Plugin}

The problem with backing up multiple servers at the same time to the
same tape library (or autoloader) is that if both servers access the
same tape drive same time, you will very likely get data corruption.
This is where the Bacula Systems shared tape storage plugin comes into play.  The
plugin ensures that only one server at a time can connect to each device
(tape drive) by using the SPC-3 SCSI reservation protocol.  Please contact
Bacula Systems support to get SAN Shared Storage Plugin specific
documentation.

\medskip
This project was funded by Bacula Systems and is available with Bacula
Enterprise Edition.

\section{Advanced Autochanger Usage}

The new \texttt{Shared Storage} Director's directive is a Bacula Enterprise
feature that allows you to share volumes between different Storage
resources. This directive should be used \textbf{only} if all \texttt{Media
  Type} are correctly set across all Devices.

The \texttt{Shared Storage} directive should be used when using the SAN
Shared Storage plugin or when accessing from the Director Storage resources
directly to Devices of an Autochanger.

When sharing volumes between different Storage resources, you will
need also to use the \texttt{reset-storageid} script before using the
\texttt{update slots} command. This script can be scheduled once a day in
an Admin job.

\begin{verbatim}
 $ /opt/bacula/scripts/reset-storageid MediaType StorageName
 $ bconsole
 * update slots storage=StorageName drive=0
\end{verbatim}

Please contact Bacula Systems support to get help on this advanced
configuration.

\medskip
This project was funded by Bacula Systems and is available with Bacula
Enterprise Edition.

\section{Enhancement of the NDMP Plugin}

The previous NDMP Plugin 4.0 was fully supporting only the NetApp hardware, the
new NDMP Plugin should now be able to support all NAS vendors with the
\texttt{volume\_format} plugin command option.

On some NDMP devices such as Celera or Blueray, the administrator can use arbitrary
volume structure name, ex:

\begin{verbatim}
 /dev/volume_home
 /rootvolume/volume_tmp
 /VG/volume_var
\end{verbatim}

The NDMP plugin should be aware of the structure organization in order to
detect if the administrator wants to restore in a new volume
(\texttt{where=/dev/vol\_tmp}) or inside a subdirectory of the targeted volume
(\texttt{where=/tmp}).

\begin{verbatim}
FileSet {
 Name = NDMPFS
 ...
 Include {
   Plugin = "ndmp:host=nasbox user=root pass=root file=/dev/vol1 volume_format=/dev/"
 }
}
\end{verbatim}

Please contact Bacula Systems support to get NDMP Plugin specific
documentation.

\medskip
This project was funded by Bacula Systems and is available with the Bacula
Enterprise Edition.

\section{Always Backup a File}

When the Accurate mode is turned on, you can decide to always backup a file
by using then new {\bf A} Accurate option in your FileSet. For example:

\begin{verbatim}
Job {
   Name = ...
   FileSet = FS_Example
   Accurate = yes
   ...
}

FileSet {
 Name = FS_Example
 Include {
   Options {
     Accurate = A
   }
   File = /file
   File = /file2
 }
 ...
}
\end{verbatim}

This project was funded by Bacula Systems based on an idea of James Harper and
is available with the Bacula Enterprise Edition.

\section{Setting Accurate Mode During at Runtime}

You are now able to specify the Accurate mode on the \texttt{run} command and
in the Schedule resource.

\begin{verbatim}
* run accurate=yes job=Test
\end{verbatim}

\begin{verbatim}
Schedule {
  Name = WeeklyCycle
  Run = Full 1st sun at 23:05
  Run = Differential accurate=yes 2nd-5th sun at 23:05
  Run = Incremental  accurate=no  mon-sat at 23:05
}
\end{verbatim}

It can allow you to save memory and and CPU resources on the catalog server in
some cases.

\medskip
These advanced tuning options are available with the Bacula Enterprise Edition.

% Common with community
\section{Additions to RunScript variables}
You can have access to JobBytes, JobFiles and Director name using \%b, \%F and \%D
in your runscript command. The Client address is now available through \%h.

\begin{verbatim}
RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%F ClientAddress=%h Dir=%D"
\end{verbatim}

\section{LZO Compression}

LZO compression was added in the Unix File Daemon. From the user point of view,
it works like the GZIP compression (just replace {\bf compression=GZIP} with
{\bf compression=LZO}).

For example:
\begin{verbatim}
Include {
   Options { compression=LZO }
   File = /home
   File = /data
}
\end{verbatim}

LZO provides much faster compression and decompression speed but lower
compression ratio than GZIP. It is a good option when you backup to disk. For
tape, the built-in compression may be a better option.

LZO is a good alternative for GZIP1 when you don't want to slow down your
backup. On a modern CPU it should be able to run almost as fast as:

\begin{itemize}
\item your client can read data from disk. Unless you have very fast disks like
  SSD or large/fast RAID array.
\item the data transfers between the file daemon and the storage daemon even on
  a 1Gb/s link.
\end{itemize}

Note that bacula only use one compression level LZO1X-1.

\medskip
The code for this feature was contributed by Laurent Papier.

\section{New Tray Monitor}

Since the old integrated Windows tray monitor doesn't work with
recent Windows versions, we have written a new Qt Tray Monitor that is available
for both Linux and Windows.  In addition to all the previous features,
this new version allows you to run Backups from 
the tray monitor menu.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=10cm]{\idir tray-monitor}
  \label{fig:traymonitor}
  \caption{New tray monitor}
\end{figure}

\begin{figure}[htbp]
  \centering
  \includegraphics[width=10cm]{\idir tray-monitor1}
  \label{fig:traymonitor1}
  \caption{Run a Job through the new tray monitor}
\end{figure}


To be able to run a job from the tray monitor, you need to
allow specific commands in the Director monitor console:
\begin{verbatim}
Console {
    Name = win2003-mon
    Password = "xxx"
    CommandACL = status, .clients, .jobs, .pools, .storage, .filesets, .messages, run
    ClientACL = *all*               # you can restrict to a specific host
    CatalogACL = *all*
    JobACL = *all*
    StorageACL = *all*
    ScheduleACL = *all*
    PoolACL = *all*
    FileSetACL = *all*
    WhereACL = *all*
}
\end{verbatim}

\medskip
This project was funded by Bacula Systems and is available with Bacula
the Enterprise Edition and the Community Edition.

\section{Purge Migration Job}

The new {\bf Purge Migration Job} directive may be added to the Migration
Job definition in the Director's configuration file. When it is enabled 
the Job that was migrated during a migration will be purged at
the end of the migration job.

For example:
\begin{verbatim}
Job {
  Name = "migrate-job"
  Type = Migrate
  Level = Full
  Client = localhost-fd
  FileSet = "Full Set"
  Messages = Standard
  Storage = DiskChanger
  Pool = Default
  Selection Type = Job
  Selection Pattern = ".*Save"
...
  Purge Migration Job = yes
}
\end{verbatim}

\medskip

This project was submitted by Dunlap Blake; testing and documentation was funded
by Bacula Systems.

\section{Changes in the Pruning Algorithm}

We rewrote the job pruning algorithm in this version. Previously, in some users
reported that the pruning process at the end of jobs was very long. It should
not be longer the case. Now, Bacula won't prune automatically a Job if this
particular Job is needed to restore data. Example:

\begin{verbatim}
JobId: 1  Level: Full
JobId: 2  Level: Incremental
JobId: 3  Level: Incremental
JobId: 4  Level: Differential
.. Other incrementals up to now
\end{verbatim}

In this example, if the Job Retention defined in the Pool or in the Client
resource causes that Jobs with Jobid in 1,2,3,4 can be pruned, Bacula will
detect that JobId 1 and 4 are essential to restore data at the current state
and will prune only JobId 2 and 3.

\texttt{Important}, this change affect only the automatic pruning step after a
Job and the \texttt{prune jobs} Bconsole command. If a volume expires after the
\texttt{VolumeRetention} period, important jobs can be pruned.

\section{Ability to Verify any specified Job}
You now have the ability to tell Bacula which Job should verify instead of
automatically verify just the last one.

This feature can be used with VolumeToCatalog, DiskToCatalog and Catalog level.

To verify a given job, just specify the Job jobid in argument when starting the
job.
\begin{verbatim}
*run job=VerifyVolume jobid=1 level=VolumeToCatalog
Run Verify job
JobName:     VerifyVolume
Level:       VolumeToCatalog
Client:      127.0.0.1-fd
FileSet:     Full Set
Pool:        Default (From Job resource)
Storage:     File (From Job resource)
Verify Job:  VerifyVol.2010-09-08_14.17.17_03
Verify List: /tmp/regress/working/VerifyVol.bsr
When:        2010-09-08 14:17:31
Priority:    10
OK to run? (yes/mod/no):
\end{verbatim}

\medskip
This project was funded by Bacula Systems and is available with Bacula
Enterprise Edition and Community Edition.