\chapter{Migration and Copy}
\label{MigrationChapter}
-\index[general]{Migration}
-\index[general]{Copy}
+\index[general]{Migration}
+\index[general]{Copy}
The term Migration, as used in the context of Bacula, means moving data from
one Volume to another. In particular it refers to a Job (similar to a backup
as a copy rather than a backup job, and hence is not directly available for
restore. If bacula founds a copy when a job record is purged (deleted) from the
catalog, it will promote the copy as \textsl{real} backup and will make it
-available for automatic restore.
+available for automatic restore.
The Copy and the Migration jobs run without using the File daemon by copying
the data from the old backup Volume to a different Volume in a different Pool.
-The section process for which Job or Jobs are migrated
+The selection process for which Job or Jobs are migrated
can be based on quite a number of different criteria such as:
-\begin{itemize}
+\begin{bsysitemize}
\item a single previous Job
\item a Volume
\item a Client
\item the time a Job has been on a Volume
\item high and low water marks (usage or occupation) of a Pool
\item Volume size
-\end{itemize}
+\end{bsysitemize}
The details of these selection criteria will be defined below.
To run a Migration job, you must first define a Job resource very similar
to a Backup Job but with {\bf Type = Migrate} instead of {\bf Type =
-Backup}. One of the key points to remember is that the Pool that is
+Backup}. One of the key points to remember is that the Pool that is
specified for the migration job is the only pool from which jobs will
be migrated, with one exception noted below. In addition, the Pool to
which the selected Job or Jobs will be migrated is defined by the {\bf
However, when doing migration, this is a very undesirable condition. For
migration to work properly, you should use Pools containing only Volumes of
the same Media Type for all migration jobs.
-
+
The migration job normally is either manually started or starts
from a Schedule much like a backup job. It searches
for a previous backup Job or Jobs that match the parameters you have
done nothing, but normally at a minimum, three jobs are involved during a
migration:
-\begin{itemize}
+\begin{bsysitemize}
\item The currently running Migration control Job. This is only
a control job for starting the migration child jobs.
\item The previous Backup Job (already run). The File records
for this Job are purged if the Migration job successfully
terminates. The original data remains on the Volume until
it is recycled and rewritten.
-\item A new Migration Backup Job that moves the data from the
+\item A new Migration Backup Job that moves the data from the
previous Backup job to the new Volume. If you subsequently
do a restore, the data will be read from this Job.
-\end{itemize}
+\end{bsysitemize}
-If the Migration control job finds a number of JobIds to migrate (e.g.
+If the Migration control job finds a number of JobIds to migrate (e.g.
it is asked to migrate one or more Volumes), it will start one new
migration backup job for each JobId found on the specified Volumes.
Please note that Migration doesn't scale too well since Migrations are
\section{Migration and Copy Job Resource Directives}
The following directives can appear in a Director's Job resource, and they
-are used to define a Migration job.
+are used to define a Migration job.
\begin{description}
\item [Pool = \lt{}Pool-name\gt{}] The Pool specified in the Migration
for finding JobIds to migrate. The exception to this is when {\bf
Selection Type = SQLQuery}, and although a Pool directive must still be
specified, no Pool is used, unless you specifically include it in the
- SQL query. Note, in any case, the Pool resource defined by the Pool
+ SQL query. Note, in any case, the Pool resource defined by the Pool
directove must contain a {\bf Next Pool = ...} directive to define the
Pool to which the data will be migrated.
If you subsequently delete a JobId that has a copy, the copy will be
automatically upgraded to a Backup rather than a Copy, and it will
subsequently be used for restoration.
-
-\item [Selection Type = \lt{}Selection-type-keyword\gt{}]
+
+\item [Selection Type = \lt{}Selection-type-keyword\gt{}]
The \lt{}Selection-type-keyword\gt{} determines how the migration job
will go about selecting what JobIds to migrate. In most cases, it is
used in conjunction with a {\bf Selection Pattern} to give you fine
database
entries. The bytes calculate for Migration is based on the value stored
in the Job records of the Jobs to be migrated. These do not include the
- Storage daemon overhead as is in the total Pool size. As a consequence,
+ Storage daemon overhead as is in the total Pool size. As a consequence,
normally, the migration will migrate more bytes than strictly necessary.
\item [PoolTime] The PoolTime selection type will cause the Migration job to
look at the time each JobId has been in the Pool since the job ended.
- All Jobs in the Pool longer than the time specified on {\bf Migration Time}
+ All Jobs in the Pool longer than the time specified on {\bf Migration Time}
directive in the Pool resource will be migrated.
\item [PoolUncopiedJobs] This selection which copies all jobs from a pool
\item [Selection Pattern = \lt{}Quoted-string\gt{}]
The Selection Patterns permitted for each Selection-type-keyword are
- described above.
+ described above.
For the OldestVolume and SmallestVolume, this
- Selection pattern is not used (ignored).
+ Selection pattern is not used (ignored).
For the Client, Volume, and Job
keywords, this pattern must be a valid regular expression that will filter
For the SQLQuery keyword, this pattern must be a valid SELECT SQL statement
that returns JobIds.
+\item [ Purge Migration Job = \lt{}yes/no\gt{}]
+ This directive may be added to the Migration Job definition in the Director
+ configuration file to purge the job migrated at the end of a migration.
+
\end{description}
\section{Migration Pool Resource Directives}
\begin{description}
\item [Migration Time = \lt{}time-specification\gt{}]
If a PoolTime migration is done, the time specified here in seconds (time
- modifiers are permitted -- e.g. hours, ...) will be used. If the
+ modifiers are permitted -- e.g. hours, ...) will be used. If the
previous Backup Job or Jobs selected have been in the Pool longer than
the specified PoolTime, then they will be migrated.
trigger a migration if a {\bf PoolOccupancy} migration selection
type has been specified. The fact that the Pool
usage goes above this level does not automatically trigger a migration
- job. However, if a migration job runs and has the PoolOccupancy selection
+ job. However, if a migration job runs and has the PoolOccupancy selection
type set, the Migration High Bytes will be applied. Bacula does not
currently restrict a pool to have only a single Media Type, so you
must keep in mind that if you mix Media Types in a Pool, the results
This directive specifies the number of bytes in the Pool which will
stop a migration if a {\bf PoolOccupancy} migration selection
type has been specified and triggered by more than Migration High
- Bytes being in the pool. In other words, once a migration job
+ Bytes being in the pool. In other words, once a migration job
is started with {\bf PoolOccupancy} migration selection and it
- determines that there are more than Migration High Bytes, the
+ determines that there are more than Migration High Bytes, the
migration job will continue to run jobs until the number of
bytes in the Pool drop to or below Migration Low Bytes.
\item [Storage = \lt{}storage-specification\gt{}]
The Storage directive specifies what Storage resource will be used
for all Jobs that use this Pool. It takes precedence over any other
- Storage specifications that may have been given such as in the
+ Storage specifications that may have been given such as in the
Schedule Run directive, or in the Job resource. We highly recommend
that you define the Storage resource to be used in the Pool rather
than elsewhere (job, schedule run, ...).
\section{Important Migration Considerations}
\index[general]{Important Migration Considerations}
-\begin{itemize}
+\begin{bsysitemize}
\item Each Pool into which you migrate Jobs or Volumes {\bf must}
- contain Volumes of only one Media Type.
+ contain Volumes of only one Media Type.
\item Migration takes place on a JobId by JobId basis. That is
each JobId is migrated in its entirety and independently
indicate that the job was migrated.
\item Jobs on Volumes will be Migration only if the Volume is
- marked, Full, Used, or Error. Volumes that are still
+ marked, Full, Used, or Error. Volumes that are still
marked Append will not be considered for migration. This
prevents Bacula from attempting to read the Volume at
the same time it is writing it. It also reduces other deadlock
\item Migration is done only when you run a Migration job. If you set a
Migration High Bytes and that number of bytes is exceeded in the Pool
- no migration job will automatically start. You must schedule the
+ no migration job will automatically start. You must schedule the
migration jobs, and they must run for any migration to take place.
\item If you migrate a number of Volumes, a very large number of Migration
- jobs may start.
+ jobs may start.
\item Figuring out what jobs will actually be migrated can be a bit complicated
- due to the flexibility provided by the regex patterns and the number of
- different options. Turning on a debug level of 100 or more will provide
- a limited amount of debug information about the migration selection
+ due to the flexibility provided by the regex patterns and the number of
+ different options. Turning on a debug level of 100 or more will provide
+ a limited amount of debug information about the migration selection
process.
\item Bacula currently does only minimal Storage conflict resolution, so you
same device or Bacula may block waiting to reserve a drive that it
will never find. In general, ensure that all your migration
pools contain only one Media Type, and that you always
- migrate to pools with different Media Types.
+ migrate to pools with different Media Types.
\item The {\bf Next Pool = ...} directive must be defined in the Pool
referenced in the Migration Job to define the Pool into which the
potential bottle neck and does not scale very well to large numbers
of jobs.
-\item Only migration of Selection Types of Job and Volume have
+\item Only migration of Selection Types of Job and Volume have
been carefully tested. All the other migration methods (time,
occupancy, smallest, oldest, ...) need additional testing.
\item Migration is only implemented for a single Storage daemon. You
cannot read on one Storage daemon and write on another.
-\end{itemize}
+\end{bsysitemize}
\section{Example Migration Jobs}
by the migration job).
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
# Define the backup Job
Job {
Name = "NightlySave"
Media Type = DLT8000 # same as MediaType in Storage daemon
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
-Where we have included only the essential information -- i.e. the
+Where we have included only the essential information -- i.e. the
Director, FileSet, Catalog, Client, Schedule, and Messages resources are
omitted.
Now, if we add the following Job resource to this conf file.
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Job {
Name = "migrate-volume"
Type = Migrate
Level = Full
- Client = rufus-fd
+ Client = rufus-fd
FileSet = "Full Set"
Messages = Standard
Pool = Default
Selection Type = Volume
Selection Pattern = "File"
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
and then run the job named {\bf migrate-volume}, all volumes in the Pool
-named Default (as specified in the migrate-volume Job that match the
-regular expression pattern {\bf File} will be migrated to tape storage
+named Default (as specified in the migrate-volume Job that match the
+regular expression pattern {\bf File} will be migrated to tape storage
DLTDrive because the {\bf Next Pool} in the Default Pool specifies that
Migrations should go to the pool named {\bf Tape}, which uses
Storage {\bf DLTDrive}.
If instead, we use a Job resource as follows:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Job {
Name = "migrate"
Type = Migrate
Messages = Standard
Pool = Default
Maximum Concurrent Jobs = 4
- Selection Type = Job
+ Selection Type = Job
Selection Pattern = ".*Save"
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
All jobs ending with the name Save will be migrated from the File Default to
projects / bacula / docs / blobdiff