\index[general]{Migration}
\addcontentsline{toc}{section}{Migration}
-The term Migration as used in the context of Bacula means moving data from one
-Volume to another, and in particular it is
-a Job (similar to a backup job) that reads data that was previously
-backed up to a Volume and writes it to another Volume purging the
-catalog records associated with the first backup job. In other words,
-Migration moves Bacula Job data from one Volume to another. Although we
-mention Volumes, in reality, Migration is based on moving individual Jobs from
-one Pool to another.
+The term Migration, as used in the context of Bacula, means moving data from
+one Volume to another. In particular it refers to a Job (similar to a backup
+job) that reads data that was previously backed up to a Volume and writes
+it to another Volume. As part of this process, the File catalog records
+associated with the first backup job are purged. In other words, Migration
+moves Bacula Job data from one Volume to another. Although we mention
+Volumes to simplify the subject, in reality, Migration reads the data
+from one Volume and writes it to different Volume in a different pool,
+which is equivalent to moving individual Jobs from one Pool to another.
Migrations can be based on quite a number of different criteria such as:
-a single previous Job; a Volume; a Client; a regular expression matching
-a Job, Volume, or Client name; the time a Job is on a Volume; high
-and low water marks (usage or occupation) of a Pool; Volume size, ...
-The details of these selection criteria will be defined below.
+\begin{itemize}
+\item a single previous Job
+\item a Volume
+\item a Client
+\item a regular expression matching a Job, Volume, or Client name
+\item the time a Job is on a Volume
+\item high and low water marks (usage or occupation) of a Pool
+\item Volume size
+\end{itemize}
+The details of these selection criteria will be defined below.
-To run a Migration job, you must have defined a Job resource very similar
+To run a Migration job, you must first define a Job resource very similar
to a Backup Job but with {\bf Type = Migrate} instead of {\bf Type =
-Backup}. The migration job then starts much like a backup job starts and
-searches for a previous backup Job or Jobs that matches the parameters you have
-specified in the migration Job resource (detailed a bit later). Then
-for each previous backup Job found, the Migration Job will run a new
-Job which copies the old Job data from the previous Volume to a new
-Volume defined in the Migration Pool. It is possible that no prior
-Jobs are found for migration, in which case, the Migration job will
-simply terminate having done nothing, but normally at a minimum, three jobs
-are involved during a migration:
+Backup}. One of the key points to remember is that the Pool that is
+specified for the migration job is the only pool from which jobs will
+be migrated, with one exception noted below.
+
+The migration job normally is either manually started or starts
+from a Schedule much like a backup job. It searches
+for a previous backup Job or Jobs that match the parameters you have
+specified in the migration Job resource, primarily a {\bf Selection Type}
+(detailed a bit later). Then for
+each previous backup JobId found, the Migration Job will run a new Job which
+copies the old Job data from the previous Volume to a new Volume in
+the Migration Pool. It is possible that no prior Jobs are found for
+migration, in which case, the Migration job will simply terminate having
+done nothing, but normally at a minimum, three jobs are involved during a
+migration:
\begin{itemize}
-\item The currently running Migration Job
-\item The previous Backup Job
-\item A new Backup Job that writes the previous data to
- the new Volume.
+\item The currently running Migration control Job
+\item The previous Backup Job (already run)
+\item A new Migration Backup Job that moves the data from the
+ previous Backup job to the new Volume.
\end{itemize}
-If the Migration is for a particular Volume, multiple new Backup Jobs
-may be started, one for each Job on the Volume.
-
-
+If the Migration control job finds a number of JobIds to migrate (e.g.
+it is asked to migrate one or more Volumes), it will start one new
+migration backup job for each JobId found.
\subsection*{Migration Job Resource Directives}
\addcontentsline{toc}{section}{Migration Job Resource Directives}
are used to define a Migration job.
\begin{description}
+\item [Pool = \lt{}Pool-name\gt{}] The Pool specified in the Migration
+ control Job is not a new directive for the Job resource, but it is
+ particularly important because it determines what Pool will be examined for
+ finding JobIds to migrate. The exception to this is when {\bf Selection
+ Type = SQLQuery}, in which case no Pool is used, unless you
+ specifically include it in the SQL query.
+
\item [Type = Migrate]
-This defines the job that is run as being a Migration Job.
-Migration Jobs do not have any Files associated with them, and in
-that sense they are more or less like an Admin job. They just
-check to see if there is anything to Migrate then possibly start
-and control new Backup jobs to move the data.
+ {\bf Migrate} is a new type that defines the job that is run as being a
+ Migration Job. A Migration Job is a sort of control job and does not have
+ any Files associated with it, and in that sense they are more or less like
+ an Admin job. Migration jobs simply check to see if there is anything to
+ Migrate then possibly start and control new Backup jobs to migrate the data
+ from the specified Pool to another Pool.
\item [Selection Type = \lt{}Selection-type-keyword\gt{}]
- Where \lt{}Selection-type-keyword\gt{} can be:
+ The \lt{}Selection-type-keyword\gt{} determines how the migration job
+ will go about selecting what JobIds to migrate. In most cases, it is
+ used in conjunction with a {\bf Selection Pattern} to give you fine
+ control over exactly what JobIds are selected. The possible values
+ for \lt{}Selection-type-keyword\gt{} are:
\begin{description}
- \item [SmallestVolume] This selection keyword selects the smallest volume in
- the Pool to be migrated.
- \item [OldestVolume] This selection keyword selects the oldest volume in
- the Pool to be migrated.
- \item [Client]
- \item [Volume]
- \item [Job]
- \item [SQLQuery]
- \item [PoolOccupancy] Not yet implemented.
- \item [PoolTime] Not yet implemented.
- \end{description}
-\end{description}
+ \item [SmallestVolume] This selection keyword selects the volume with the
+ fewest bytes from the Pool to be migrated. The Pool to be migrated
+ is the Pool defined in the Migration Job resource. The migration
+ control job will then start and run one migration backup job for
+ each of the Jobs found on this Volume. The Selection Pattern, if
+ specified, is not used.
+
+ \item [OldestVolume] This selection keyword selects the volume with the
+ oldest last write time in the Pool to be migrated. The Pool to be
+ migrated is the Pool defined in the Migration Job resource. The
+ migration control job will then start and run one migration backup
+ job for each of the Jobs found on this Volume. The Selection
+ Pattern, if specified, is not used.
+
+ \item [Client] The Client selection type, first selects all the Clients
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Client names, giving
+ a filtered Client name list. All jobs that were backed up for those
+ filtered (regexed) Clients will be migrated.
+ The migration control job will then start and run one migration
+ backup job for each of the JobIds found for those filtered Clients.
+
+ \item [Volume] The Volume selection type, first selects all the Volumes
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Volume names, giving
+ a filtered Volume list. All JobIds that were backed up for those
+ filtered (regexed) Volumes will be migrated.
+ The migration control job will then start and run one migration
+ backup job for each of the JobIds found on those filtered Volumes.
+
+ \item [Job] The Job selection type, first selects all the Jobs (as
+ defined on the {\bf Name} directive in a Job resource)
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Job names, giving
+ a filtered Job name list. All JobIds that were run for those
+ filtered (regexed) Job names will be migrated. Note, for a given
+ Job named, they can be many jobs (JobIds) that ran.
+ The migration control job will then start and run one migration
+ backup job for each of the Jobs found.
+
+ \item [SQLQuery] The SQLQuery selection type, used the {\bf Selection
+ Pattern} as an SQL query to obtain the JobIds to be migrated.
+ The Selection Pattern must be a valid SELECT SQL statement for your
+ SQL engine, and it must return the JobId as the first field
+ of the SELECT.
+ \item [PoolOccupancy] This selection type will cause the Migration job
+ to compute the total size of the specified pool for all Media Types
+ combined. If it exceeds the {\bf Migration High Bytes} defined in
+ the Pool, the Migration job will migrate all JobIds beginning with
+ the oldest Volume in the pool (determined by Last Write time) until
+ the Pool bytes drop below the {\bf Migration Low Bytes} defined in the
+ Pool. This calculation should be consider rather approximative because
+ it is made once by the Migration job before migration is begun, and
+ thus does not take into account additional data written into the Pool
+ during the migration. In addition, the calculation of the total Pool
+ byte size is based on the Volume bytes saved in the Volume (Media)
+database
+ entries. The bytes caculate for Migration is based on the value stored
+ in the Job records of the Jobs to be migrated. These do not include the
+ Storage daemon overhead as is in the total Pool size. As a consequence,
+ normally, the migration will migrate more bytes than strictly necessary.
+
+ \item [PoolTime] The PoolTime selection type will cause the Migration job to
+ look at the time each JobId has been in the Pool since the job ended.
+ All Jobs in the Pool longer than the time specified on {\bf Migration Time}
+ directive in the Pool resource will be migrated.
+ \end{description}
\item [Selection Pattern = \lt{}Quoted-string\gt{}]
-The Selection Patterns permitted for each Selection-type-keyword
-are described above.
+ The Selection Patterns permitted for each Selection-type-keyword are
+ described above.
+
+ For the OldestVolume and SmallestVolume, this
+ Selection pattern is not used (ignored).
+
+ For the Client, Volume, and Job
+ keywords, this pattern must be a valid regular expression that will filter
+ the appropriate item names found in the Pool.
+
+ For the SQLQuery keyword, this pattern must be a valid SELECT SQL statement
+ that returns JobIds.
\end{description}
\addcontentsline{toc}{section}{Migration Pool Resource Directives}
The following directives can appear in a Director's Pool resource, and they
-are used to define a Migration job.
+are used to define a Migration job.
\begin{description}
\item [Migration Time = \lt{}time-specification\gt{}]
-If a PoolTime migration is done, the time specified here in seconds (time
-modifiers are permitted -- e.g. hours, ...) will be used. If the
-previous Backup Job or Jobs selected have been in the Pool longer than
-the specified PoolTime, then they will be migrated.
-
+ If a PoolTime migration is done, the time specified here in seconds (time
+ modifiers are permitted -- e.g. hours, ...) will be used. If the
+ previous Backup Job or Jobs selected have been in the Pool longer than
+ the specified PoolTime, then they will be migrated.
+
\item [Migration High Bytes = \lt{}byte-specification\gt{}]
+ This directive specifies the number of bytes in the Pool which will
+ trigger a migration if a {\bf PoolOccupancy} migration selection
+ type has been specified. The fact that the Pool
+ usage goes above this level does not automatically trigger a migration
+ job. However, if a migration job runs and has the PoolOccupancy selection
+ type set, the Migration High Bytes will be applied. Bacula does not
+ currently restrict a pool to have only a single Media Type, so you
+ must keep in mind that if you mix Media Types in a Pool, the results
+ may not be what you want, as the Pool count of all bytes will be
+ for all Media Types combined.
\item [Migration Low Bytes = \lt{}byte-specification\gt{}]
+ This directive specifies the number of bytes in the Pool which will
+ stop a migration if a {\bf PoolOccupancy} migration selection
+ type has been specified and triggered by more than Migration High
+ Bytes being in the pool. In other words, once a migration job
+ is started with {\bf PoolOccupancy} migration selection and it
+ determines that there are more than Migration High Bytes, the
+ migration job will continue to run jobs until the number of
+ bytes in the Pool drop to or below Migration Low Bytes.
\item [Next Pool = \lt{}pool-specification\gt{}]
+ The Next Pool directive specifies the pool to which Jobs will be
+ migrated.
\item [Storage = \lt{}storage-specification\gt{}]
-
+ The Storage directive specifies what Storage resource will be used
+ for all Jobs that use this Pool. It takes precedence over any other
+ Storage specifications that may have been given such as in the
+ Schedule Run directive, or in the Job resource.
\end{description}
+
+\subsection*{Important Migration Considerations}
+\index[general]{Important Migration Considerations}
+\addcontentsline{toc}{subsection}{Important Migration Considerations}
+\begin{itemize}
+\item Migration takes place on a JobId by JobId basis. That is
+ each JobId is migrated in its entirety and independently
+ of other JobIds. Once the Job is migrated, it will be
+ on the new medium in the new Pool, but for the most part,
+ aside from having a new JobId, it will appear with all the
+ same characteristics of the original job (start, end time, ...).
+ The column RealEndTime in the Job table will contain the
+ time and date that the Migration terminated, and by comparing
+ it with the EndTime column you can tell whether or not the
+ job was migrated. The original job is purged of its File
+ records, and its Type field is changed from "B" to "M" to
+ indicate that the job was migrated.
+
+\item Jobs on Volumes will be Migration only if the Volume is
+ marked, Full, Used, or Error. Volumes that are still
+ marked Append will not be considered for migration. This
+ prevents Bacula from attempting to read the Volume at
+ the same time it is writing it.
+
+\item As noted above, for the Migration High Bytes, the calculation
+ of the bytes to migrate is somewhat approximate.
+
+\item If you keep Volumes of different Media Types in the same Pool,
+ it is not clear how well migration will work. We recommend only
+ one Media Type per pool.
+
+\item It is possible to get into a resource deadlock where Bacula does
+ not find enough drives to simultaneously read and write all the
+ Volumes needed to do Migrations. For the moment, you must take
+ care as all the resource deadlock algorithms are not yet implemented.
+
+\item Migration is done only when you run a Migration job. If you set a
+ Migration High Bytes and that number of bytes is exceeded in the Pool
+ no migration job will automatically start. You must schedule the
+ migration jobs yourself.
+
+\item If you migrate a number of Volumes, a very large number of Migration
+ jobs may start.
+
+\item Figuring out what jobs will actually be migrated can be a bit complicated
+ due to the flexibility provided by the regex patterns and the number of
+ different options. Turning on a debug level of 100 or more will provide
+ a limited amount of debug information about the migration selection
+ process.
+\end{itemize}
+
+
+\subsection*{Example Migration Jobs}
+\index[general]{Example Migration Jobs}
+\addcontentsline{toc}{subsection}{Example Migration Jobs}
+
+When you specify a Migration Job, you must specify all the standard
+directives as for a Job. However, certain such as the Level, Client, and
+FileSet, though they must be defined, are ignored by the Migration job
+because the values from the original job used instead.
+
+As an example, suppose you have the following Job that
+you run every night:
+
+\footnotesize
+\begin{verbatim}
+# Define the backup Job
+Job {
+ Name = "NightlySave"
+ Type = Backup
+ Level = Incremental # default
+ Client=rufus-fd
+ FileSet="Full Set"
+ Schedule = "WeeklyCycle"
+ Messages = Standard
+ Pool = Default
+}
+
+# Default pool definition
+Pool {
+ Name = Default
+ Pool Type = Backup
+ AutoPrune = yes
+ Recycle = yes
+ Next Pool = Tape
+ Storage = File
+ LabelFormat = "File"
+}
+
+# Tape pool definition
+Pool {
+ Name = Tape
+ Pool Type = Backup
+ AutoPrune = yes
+ Recycle = yes
+ Storage = DLTDrive
+}
+
+# Definition of File storage device
+Storage {
+ Name = File
+ Address = rufus
+ Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
+ Device = "File" # same as Device in Storage daemon
+ Media Type = File # same as MediaType in Storage daemon
+}
+
+# Definition of DLT tape storage device
+Storage {
+ Name = DLTDrive
+ Address = rufus
+ Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
+ Device = "HP DLT 80" # same as Device in Storage daemon
+ Media Type = DLT8000 # same as MediaType in Storage daemon
+}
+
+\end{verbatim}
+\normalsize
+
+Where we have included only the essential information -- i.e. the
+Director, FileSet, Catalog, Client, Schedule, and Messages resources are
+omitted.
+
+As you can see, by running the NightlySave Job, the data will be backed up
+to File storage using the Default pool to specify the Storage as File.
+
+Now, if we add the following Job resource to this conf file.
+
+\footnotesize
+\begin{verbatim}
+Job {
+ Name = "migrate-volume"
+ Type = Migrate
+ Level = Full
+ Client = rufus-fd
+ FileSet = "Full Set"
+ Messages = Standard
+ Storage = DLTDrive
+ Pool = Default
+ Maximum Concurrent Jobs = 4
+ Selection Type = Volume
+ Selection Pattern = "File"
+}
+\end{verbatim}
+\normalsize
+
+and then run the job named {\bf migrate-volume}, all volumes in the Pool
+named Default (as specified in the migrate-volume Job that match the
+regular expression pattern {\bf File} will be migrated to tape storage
+DLTDrive because the {\bf Next Pool} in the Default Pool specifies that
+Migrations should go to the pool named {\bf Tape}, which uses
+Storage {\bf DLTDrive}.
+
+If instead, we use a Job resource as follows:
+
+\footnotesize
+\begin{verbatim}
+Job {
+ Name = "migrate"
+ Type = Migrate
+ Level = Full
+ Client = rufus-fd
+ FileSet="Full Set"
+ Messages = Standard
+ Storage = DLTDrive
+ Pool = Default
+ Maximum Concurrent Jobs = 4
+ Selection Type = Job
+ Selection Pattern = ".*Save"
+}
+\end{verbatim}
+\normalsize
+
+All jobs ending with the name Save will be migrated from the File Default to
+the Tape Pool, or from File storage to Tape storage.
projects / bacula / docs / blobdiff