The term Migration, as used in the context of Bacula, means moving data from
one Volume to another. In particular it refers to a Job (similar to a backup
job) that reads data that was previously backed up to a Volume and writes
-it to another Volume. As part of this process, catalog records associated with the first
-backup job are purged. In other words, Migration moves Bacula Job data from one
-Volume to another. Although we mention Volumes to simplify the subject,
-in reality, Migration is based upon the concept of moving individual Jobs from one Pool to another.
+it to another Volume. As part of this process, the File catalog records
+associated with the first backup job are purged. In other words, Migration
+moves Bacula Job data from one Volume to another. Although we mention
+Volumes to simplify the subject, in reality, Migration reads the data
+from one Volume and writes it to different Volume in a different pool,
+which is equivalent to moving individual Jobs from one Pool to another.
Migrations can be based on quite a number of different criteria such as:
\begin{itemize}
The migration job normally is either manually started or starts
from a Schedule much like a backup job. It searches
for a previous backup Job or Jobs that match the parameters you have
-specified in the migration Job resource, primiarily a {\bf Selection Type}
+specified in the migration Job resource, primarily a {\bf Selection Type}
(detailed a bit later). Then for
each previous backup JobId found, the Migration Job will run a new Job which
copies the old Job data from the previous Volume to a new Volume in
\begin{description}
\item [Pool = \lt{}Pool-name\gt{}] The Pool specified in the Migration
-control Job is not a new directive for the Job resource, but it is
-particularly important because it determines what Pool will be examined for
-finding JobIds to migrate. The exception to this is when {\bf Selection
-Type = SQLQuery}, in which case no Pool is used, unless you
-specifically include it in the SQL query.
+ control Job is not a new directive for the Job resource, but it is
+ particularly important because it determines what Pool will be examined for
+ finding JobIds to migrate. The exception to this is when {\bf Selection
+ Type = SQLQuery}, in which case no Pool is used, unless you
+ specifically include it in the SQL query.
\item [Type = Migrate]
-{\bf Migrate} is a new type that defines the job that is run as being a
-Migration Job. A Migration Job is a sort of control job and does not have
-any Files associated with it, and in that sense they are more or less like
-an Admin job. Migration jobs simply check to see if there is anything to Migrate then
-possibly start and control new Backup jobs to migrate the data from the
-specified Pool to another Pool.
+ {\bf Migrate} is a new type that defines the job that is run as being a
+ Migration Job. A Migration Job is a sort of control job and does not have
+ any Files associated with it, and in that sense they are more or less like
+ an Admin job. Migration jobs simply check to see if there is anything to
+ Migrate then possibly start and control new Backup jobs to migrate the data
+ from the specified Pool to another Pool.
\item [Selection Type = \lt{}Selection-type-keyword\gt{}]
The \lt{}Selection-type-keyword\gt{} determines how the migration job
control job will then start and run one migration backup job for
each of the Jobs found on this Volume. The Selection Pattern, if
specified, is not used.
+
\item [OldestVolume] This selection keyword selects the volume with the
oldest last write time in the Pool to be migrated. The Pool to be
migrated is the Pool defined in the Migration Job resource. The
migration control job will then start and run one migration backup
job for each of the Jobs found on this Volume. The Selection
Pattern, if specified, is not used.
+
\item [Client] The Client selection type, first selects all the Clients
that have been backed up in the Pool specified by the Migration
Job resource, then it applies the {\bf Selection Pattern} (defined
filtered (regexed) Clients will be migrated.
The migration control job will then start and run one migration
backup job for each of the JobIds found for those filtered Clients.
+
\item [Volume] The Volume selection type, first selects all the Volumes
that have been backed up in the Pool specified by the Migration
Job resource, then it applies the {\bf Selection Pattern} (defined
filtered (regexed) Volumes will be migrated.
The migration control job will then start and run one migration
backup job for each of the JobIds found on those filtered Volumes.
+
\item [Job] The Job selection type, first selects all the Jobs (as
defined on the {\bf Name} directive in a Job resource)
that have been backed up in the Pool specified by the Migration
Job named, they can be many jobs (JobIds) that ran.
The migration control job will then start and run one migration
backup job for each of the Jobs found.
+
\item [SQLQuery] The SQLQuery selection type, used the {\bf Selection
Pattern} as an SQL query to obtain the JobIds to be migrated.
The Selection Pattern must be a valid SELECT SQL statement for your
SQL engine, and it must return the JobId as the first field
of the SELECT.
- \item [PoolOccupancy] Not yet implemented.
- \item [PoolTime] Not yet implemented.
+
+ \item [PoolOccupancy] This selection type will cause the Migration job
+ to compute the total size of the specified pool for all Media Types
+ combined. If it exceeds the {\bf Migration High Bytes} defined in
+ the Pool, the Migration job will migrate all JobIds beginning with
+ the oldest Volume in the pool (determined by Last Write time) until
+ the Pool bytes drop below the {\bf Migration Low Bytes} defined in the
+ Pool. This calculation should be consider rather approximative because
+ it is made once by the Migration job before migration is begun, and
+ thus does not take into account additional data written into the Pool
+ during the migration. In addition, the calculation of the total Pool
+ byte size is based on the Volume bytes saved in the Volume (Media)
+database
+ entries. The bytes caculate for Migration is based on the value stored
+ in the Job records of the Jobs to be migrated. These do not include the
+ Storage daemon overhead as is in the total Pool size. As a consequence,
+ normally, the migration will migrate more bytes than strictly necessary.
+
+ \item [PoolTime] The PoolTime selection type will cause the Migration job to
+ look at the time each JobId has been in the Pool since the job ended.
+ All Jobs in the Pool longer than the time specified on {\bf Migration Time}
+ directive in the Pool resource will be migrated.
\end{description}
\item [Selection Pattern = \lt{}Quoted-string\gt{}]
\addcontentsline{toc}{section}{Migration Pool Resource Directives}
The following directives can appear in a Director's Pool resource, and they
-are used to define a Migration job.
+are used to define a Migration job.
\begin{description}
\item [Migration Time = \lt{}time-specification\gt{}]
-If a PoolTime migration is done, the time specified here in seconds (time
-modifiers are permitted -- e.g. hours, ...) will be used. If the
-previous Backup Job or Jobs selected have been in the Pool longer than
-the specified PoolTime, then they will be migrated.
-
+ If a PoolTime migration is done, the time specified here in seconds (time
+ modifiers are permitted -- e.g. hours, ...) will be used. If the
+ previous Backup Job or Jobs selected have been in the Pool longer than
+ the specified PoolTime, then they will be migrated.
+
\item [Migration High Bytes = \lt{}byte-specification\gt{}]
This directive specifies the number of bytes in the Pool which will
trigger a migration if a {\bf PoolOccupancy} migration selection
- type has been specified. Please note, that the fact that the Pool
+ type has been specified. The fact that the Pool
usage goes above this level does not automatically trigger a migration
- job.
+ job. However, if a migration job runs and has the PoolOccupancy selection
+ type set, the Migration High Bytes will be applied. Bacula does not
+ currently restrict a pool to have only a single Media Type, so you
+ must keep in mind that if you mix Media Types in a Pool, the results
+ may not be what you want, as the Pool count of all bytes will be
+ for all Media Types combined.
\item [Migration Low Bytes = \lt{}byte-specification\gt{}]
This directive specifies the number of bytes in the Pool which will
for all Jobs that use this Pool. It takes precedence over any other
Storage specifications that may have been given such as in the
Schedule Run directive, or in the Job resource.
-
\end{description}
+\subsection*{Important Migration Considerations}
+\index[general]{Important Migration Considerations}
+\addcontentsline{toc}{subsection}{Important Migration Considerations}
+\begin{itemize}
+\item Migration takes place on a JobId by JobId basis. That is
+ each JobId is migrated in its entirety and independently
+ of other JobIds. Once the Job is migrated, it will be
+ on the new medium in the new Pool, but for the most part,
+ aside from having a new JobId, it will appear with all the
+ same characteristics of the original job (start, end time, ...).
+ The column RealEndTime in the Job table will contain the
+ time and date that the Migration terminated, and by comparing
+ it with the EndTime column you can tell whether or not the
+ job was migrated. The original job is purged of its File
+ records, and its Type field is changed from "B" to "M" to
+ indicate that the job was migrated.
+
+\item Jobs on Volumes will be Migration only if the Volume is
+ marked, Full, Used, or Error. Volumes that are still
+ marked Append will not be considered for migration. This
+ prevents Bacula from attempting to read the Volume at
+ the same time it is writing it.
+
+\item As noted above, for the Migration High Bytes, the calculation
+ of the bytes to migrate is somewhat approximate.
+
+\item If you keep Volumes of different Media Types in the same Pool,
+ it is not clear how well migration will work. We recommend only
+ one Media Type per pool.
+
+\item It is possible to get into a resource deadlock where Bacula does
+ not find enough drives to simultaneously read and write all the
+ Volumes needed to do Migrations. For the moment, you must take
+ care as all the resource deadlock algorithms are not yet implemented.
+
+\item Migration is done only when you run a Migration job. If you set a
+ Migration High Bytes and that number of bytes is exceeded in the Pool
+ no migration job will automatically start. You must schedule the
+ migration jobs yourself.
+
+\item If you migrate a number of Volumes, a very large number of Migration
+ jobs may start.
+
+\item Figuring out what jobs will actually be migrated can be a bit complicated
+ due to the flexibility provided by the regex patterns and the number of
+ different options. Turning on a debug level of 100 or more will provide
+ a limited amount of debug information about the migration selection
+ process.
+\end{itemize}
+
+
\subsection*{Example Migration Jobs}
\index[general]{Example Migration Jobs}
\addcontentsline{toc}{subsection}{Example Migration Jobs}
+When you specify a Migration Job, you must specify all the standard
+directives as for a Job. However, certain such as the Level, Client, and
+FileSet, though they must be defined, are ignored by the Migration job
+because the values from the original job used instead.
+
As an example, suppose you have the following Job that
you run every night:
\normalsize
Where we have included only the essential information -- i.e. the
-Director, FileSet, Catalog, Client, Schedule, and Messages resources are omitted.
+Director, FileSet, Catalog, Client, Schedule, and Messages resources are
+omitted.
As you can see, by running the NightlySave Job, the data will be backed up
to File storage using the Default pool to specify the Storage as File.
projects / bacula / docs / commitdiff