Update migration chapter

[bacula/docs] / docs / manual / migration.tex

\section*{Migration}
\label{_MigrationChapter}
\index[general]{Migration} 
\addcontentsline{toc}{section}{Migration}

The term Migration, as used in the context of Bacula, means moving data from
one Volume to another.  In particular it refers to a Job (similar to a backup
job) that reads data that was previously backed up to a Volume and writes
it to another Volume.  As part of this process, the File catalog records
associated with the first backup job are purged.  In other words, Migration
moves Bacula Job data from one Volume to another.  Although we mention
Volumes to simplify the subject, in reality, Migration reads the data
from one Volume and writes it to different Volume in a different pool,
which is equivalent to moving individual Jobs from one Pool to another.

Migrations can be based on quite a number of different criteria such as:
\begin{itemize} 
\item a single previous Job
\item a Volume
\item a Client
\item a regular expression matching a Job, Volume, or Client name
\item the time a Job is on a Volume
\item high and low water marks (usage or occupation) of a Pool
\item Volume size
\end{itemize}

The details of these selection criteria will be defined below.

To run a Migration job, you must first define a Job resource very similar
to a Backup Job but with {\bf Type = Migrate} instead of {\bf Type =
Backup}.  One of the key points to remember is that the Pool that is 
specified for the migration job is the only pool from which jobs will
be migrated, with one exception noted below.
       
The migration job normally is either manually started or starts
from a Schedule much like a backup job. It searches
for a previous backup Job or Jobs that match the parameters you have
specified in the migration Job resource, primarily a {\bf Selection Type}
(detailed a bit later).  Then for
each previous backup JobId found, the Migration Job will run a new Job which
copies the old Job data from the previous Volume to a new Volume in
the Migration Pool.  It is possible that no prior Jobs are found for
migration, in which case, the Migration job will simply terminate having
done nothing, but normally at a minimum, three jobs are involved during a
migration:

\begin{itemize}
\item The currently running Migration control Job
\item The previous Backup Job (already run)
\item A new Migration Backup Job that moves the data from the 
      previous Backup job to the new Volume.
\end{itemize}

If the Migration control job finds a number of JobIds to migrate (e.g. 
it is asked to migrate one or more Volumes), it will start one new
migration backup job for each JobId found. 

\subsection*{Migration Job Resource Directives}
\addcontentsline{toc}{section}{Migration Job Resource Directives}

The following directives can appear in a Director's Job resource, and they
are used to define a Migration job.          

\begin{description}
\item [Pool = \lt{}Pool-name\gt{}] The Pool specified in the Migration
   control Job is not a new directive for the Job resource, but it is
   particularly important because it determines what Pool will be examined for
   finding JobIds to migrate.  The exception to this is when {\bf Selection
   Type = SQLQuery}, in which case no Pool is used, unless you
   specifically include it in the SQL query.

\item [Type = Migrate]
   {\bf Migrate} is a new type that defines the job that is run as being a
   Migration Job.  A Migration Job is a sort of control job and does not have
   any Files associated with it, and in that sense they are more or less like
    an Admin job.  Migration jobs simply check to see if there is anything to
   Migrate then possibly start and control new Backup jobs to migrate the data
   from the specified Pool to another Pool.
 
\item [Selection Type = \lt{}Selection-type-keyword\gt{}]  
  The \lt{}Selection-type-keyword\gt{} determines how the migration job
  will go about selecting what JobIds to migrate. In most cases, it is
  used in conjunction with a {\bf Selection Pattern} to give you fine
  control over exactly what JobIds are selected.  The possible values
  for \lt{}Selection-type-keyword\gt{} are:
  \begin{description}
  \item [SmallestVolume] This selection keyword selects the volume with the
        fewest bytes from the Pool to be migrated.  The Pool to be migrated
        is the Pool defined in the Migration Job resource.  The migration
        control job will then start and run one migration backup job for
        each of the Jobs found on this Volume.  The Selection Pattern, if
        specified, is not used.

  \item [OldestVolume] This selection keyword selects the volume with the
        oldest last write time in the Pool to be migrated.  The Pool to be
        migrated is the Pool defined in the Migration Job resource.  The
        migration control job will then start and run one migration backup
        job for each of the Jobs found on this Volume.  The Selection
        Pattern, if specified, is not used.

  \item [Client] The Client selection type, first selects all the Clients
        that have been backed up in the Pool specified by the Migration
        Job resource, then it applies the {\bf Selection Pattern} (defined
        below) as a regular expression to the list of Client names, giving
        a filtered Client name list.  All jobs that were backed up for those
        filtered (regexed) Clients will be migrated.
        The migration control job will then start and run one migration
        backup job for each of the JobIds found for those filtered Clients.

  \item [Volume] The Volume selection type, first selects all the Volumes
        that have been backed up in the Pool specified by the Migration
        Job resource, then it applies the {\bf Selection Pattern} (defined
        below) as a regular expression to the list of Volume names, giving
        a filtered Volume list.  All JobIds that were backed up for those
        filtered (regexed) Volumes will be migrated.
        The migration control job will then start and run one migration
        backup job for each of the JobIds found on those filtered Volumes.

  \item [Job] The Job selection type, first selects all the Jobs (as
        defined on the {\bf Name} directive in a Job resource)
        that have been backed up in the Pool specified by the Migration
        Job resource, then it applies the {\bf Selection Pattern} (defined
        below) as a regular expression to the list of Job names, giving
        a filtered Job name list.  All JobIds that were run for those
        filtered (regexed) Job names will be migrated.  Note, for a given
        Job named, they can be many jobs (JobIds) that ran.
        The migration control job will then start and run one migration
        backup job for each of the Jobs found.

  \item [SQLQuery] The SQLQuery selection type, used the {\bf Selection
        Pattern} as an SQL query to obtain the JobIds to be migrated.
        The Selection Pattern must be a valid SELECT SQL statement for your
        SQL engine, and it must return the JobId as the first field
        of the SELECT.

  \item [PoolOccupancy] This selection type will cause the Migration job
        to compute the total size of the specified pool for all Media Types
        combined. If it exceeds the {\bf Migration High Bytes} defined in
        the Pool, the Migration job will migrate all JobIds beginning with
        the oldest Volume in the pool (determined by Last Write time) until
        the Pool bytes drop below the {\bf Migration Low Bytes} defined in the
        Pool. This calculation should be consider rather approximative because
        it is made once by the Migration job before migration is begun, and
        thus does not take into account additional data written into the Pool
        during the migration.  In addition, the calculation of the total Pool
        byte size is based on the Volume bytes saved in the Volume (Media)
database
        entries. The bytes caculate for Migration is based on the value stored
        in the Job records of the Jobs to be migrated. These do not include the
        Storage daemon overhead as is in the total Pool size. As a consequence, 
        normally, the migration will migrate more bytes than strictly necessary.

  \item [PoolTime] The PoolTime selection type will cause the Migration job to
        look at the time each JobId has been in the Pool since the job ended.
        All Jobs in the Pool longer than the time specified on {\bf Migration Time} 
        directive in the Pool resource will be migrated.
  \end{description}

\item [Selection Pattern = \lt{}Quoted-string\gt{}]
  The Selection Patterns permitted for each Selection-type-keyword are
  described above.  

  For the OldestVolume and SmallestVolume, this
  Selection pattern is not used (ignored).  

  For the Client, Volume, and Job
  keywords, this pattern must be a valid regular expression that will filter
  the appropriate item names found in the Pool.

  For the SQLQuery keyword, this pattern must be a valid SELECT SQL statement
  that returns JobIds.

\end{description}

\subsection*{Migration Pool Resource Directives}
\addcontentsline{toc}{section}{Migration Pool Resource Directives}

The following directives can appear in a Director's Pool resource, and they
are used to define a Migration job.

\begin{description}
\item [Migration Time = \lt{}time-specification\gt{}]
   If a PoolTime migration is done, the time specified here in seconds (time
   modifiers are permitted -- e.g. hours, ...) will be used. If the     
   previous Backup Job or Jobs selected have been in the Pool longer than
   the specified PoolTime, then they will be migrated.

\item [Migration High Bytes =  \lt{}byte-specification\gt{}]
   This directive specifies the number of bytes in the Pool which will
   trigger a migration if a {\bf PoolOccupancy} migration selection
   type has been specified. The fact that the Pool
   usage goes above this level does not automatically trigger a migration
   job. However, if a migration job runs and has the PoolOccupancy selection 
   type set, the Migration High Bytes will be applied.  Bacula does not
   currently restrict a pool to have only a single Media Type, so you
   must keep in mind that if you mix Media Types in a Pool, the results
   may not be what you want, as the Pool count of all bytes will be
   for all Media Types combined.

\item [Migration Low Bytes = \lt{}byte-specification\gt{}]
   This directive specifies the number of bytes in the Pool which will
   stop a migration if a {\bf PoolOccupancy} migration selection
   type has been specified and triggered by more than Migration High
   Bytes being in the pool. In other words, once a migration job 
   is started with {\bf PoolOccupancy} migration selection and it
   determines that there are more than Migration High Bytes, the    
   migration job will continue to run jobs until the number of
   bytes in the Pool drop to or below Migration Low Bytes.

\item [Next Pool = \lt{}pool-specification\gt{}]
   The Next Pool directive specifies the pool to which Jobs will be
   migrated.

\item [Storage = \lt{}storage-specification\gt{}]
   The Storage directive specifies what Storage resource will be used
   for all Jobs that use this Pool. It takes precedence over any other
   Storage specifications that may have been given such as in the 
   Schedule Run directive, or in the Job resource.
\end{description}

\subsection*{Important Migration Considerations}
\index[general]{Important Migration Considerations}
\addcontentsline{toc}{subsection}{Important Migration Considerations}
\begin{itemize}
\item Migration takes place on a JobId by JobId basis. That is
      each JobId is migrated in its entirety and independently
      of other JobIds. Once the Job is migrated, it will be
      on the new medium in the new Pool, but for the most part,
      aside from having a new JobId, it will appear with all the
      same characteristics of the original job (start, end time, ...).
      The column RealEndTime in the Job table will contain the
      time and date that the Migration terminated, and by comparing
      it with the EndTime column you can tell whether or not the
      job was migrated.  The original job is purged of its File
      records, and its Type field is changed from "B" to "M" to
      indicate that the job was migrated.

\item Jobs on Volumes will be Migration only if the Volume is
      marked, Full, Used, or Error.  Volumes that are still 
      marked Append will not be considered for migration. This
      prevents Bacula from attempting to read the Volume at
      the same time it is writing it.

\item As noted above, for the Migration High Bytes, the calculation
      of the bytes to migrate is somewhat approximate.

\item If you keep Volumes of different Media Types in the same Pool,
      it is not clear how well migration will work.  We recommend only
      one Media Type per pool.

\item It is possible to get into a resource deadlock where Bacula does
      not find enough drives to simultaneously read and write all the
      Volumes needed to do Migrations. For the moment, you must take
      care as all the resource deadlock algorithms are not yet implemented.

\item Migration is done only when you run a Migration job. If you set a
      Migration High Bytes and that number of bytes is exceeded in the Pool
      no migration job will automatically start.  You must schedule the 
      migration jobs yourself.

\item If you migrate a number of Volumes, a very large number of Migration
      jobs may start. 

\item Figuring out what jobs will actually be migrated can be a bit complicated
      due to the flexibility provided by the regex patterns and the number of 
      different options.  Turning on a debug level of 100 or more will provide 
      a limited amount of debug information about the migration selection 
      process.
\end{itemize}


\subsection*{Example Migration Jobs}
\index[general]{Example Migration Jobs}
\addcontentsline{toc}{subsection}{Example Migration Jobs}

When you specify a Migration Job, you must specify all the standard
directives as for a Job.  However, certain such as the Level, Client, and
FileSet, though they must be defined, are ignored by the Migration job
because the values from the original job used instead.

As an example, suppose you have the following Job that
you run every night:

\footnotesize
\begin{verbatim}
# Define the backup Job
Job {
  Name = "NightlySave"
  Type = Backup
  Level = Incremental                 # default
  Client=rufus-fd
  FileSet="Full Set"
  Schedule = "WeeklyCycle"
  Messages = Standard
  Pool = Default
}

# Default pool definition
Pool {
  Name = Default
  Pool Type = Backup
  AutoPrune = yes
  Recycle = yes
  Next Pool = Tape
  Storage = File
  LabelFormat = "File"
}

# Tape pool definition
Pool {
  Name = Tape
  Pool Type = Backup
  AutoPrune = yes
  Recycle = yes
  Storage = DLTDrive
}

# Definition of File storage device
Storage {
  Name = File
  Address = rufus
  Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
  Device = "File"          # same as Device in Storage daemon
  Media Type = File        # same as MediaType in Storage daemon
}

# Definition of DLT tape storage device
Storage {
  Name = DLTDrive
  Address = rufus
  Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
  Device = "HP DLT 80"      # same as Device in Storage daemon
  Media Type = DLT8000      # same as MediaType in Storage daemon
}

\end{verbatim}
\normalsize

Where we have included only the essential information -- i.e. the 
Director, FileSet, Catalog, Client, Schedule, and Messages resources are
omitted.

As you can see, by running the NightlySave Job, the data will be backed up
to File storage using the Default pool to specify the Storage as File.

Now, if we add the following Job resource to this conf file.

\footnotesize
\begin{verbatim}
Job {
  Name = "migrate-volume"
  Type = Migrate
  Level = Full
  Client = rufus-fd 
  FileSet = "Full Set"
  Messages = Standard
  Storage = DLTDrive
  Pool = Default
  Maximum Concurrent Jobs = 4
  Selection Type = Volume
  Selection Pattern = "File"
}
\end{verbatim}
\normalsize

and then run the job named {\bf migrate-volume}, all volumes in the Pool
named Default (as specified in the migrate-volume Job that match the 
regular expression pattern {\bf File} will be migrated to tape storage 
DLTDrive because the {\bf Next Pool} in the Default Pool specifies that
Migrations should go to the pool named {\bf Tape}, which uses
Storage {\bf DLTDrive}.

If instead, we use a Job resource as follows:

\footnotesize
\begin{verbatim}
Job {
  Name = "migrate"
  Type = Migrate
  Level = Full
  Client = rufus-fd
  FileSet="Full Set"
  Messages = Standard
  Storage = DLTDrive
  Pool = Default
  Maximum Concurrent Jobs = 4
  Selection Type = Job 
  Selection Pattern = ".*Save"
}
\end{verbatim}
\normalsize

All jobs ending with the name Save will be migrated from the File Default to
the Tape Pool, or from File storage to Tape storage.