--- /dev/null
+
+\chapter{Migration and Copy}
+\label{MigrationChapter}
+\index[general]{Migration}
+\index[general]{Copy}
+
+The term Migration, as used in the context of Bacula, means moving data from
+one Volume to another. In particular it refers to a Job (similar to a backup
+job) that reads data that was previously backed up to a Volume and writes
+it to another Volume. As part of this process, the File catalog records
+associated with the first backup job are purged. In other words, Migration
+moves Bacula Job data from one Volume to another by reading the Job data
+from the Volume it is stored on, writing it to a different Volume in a
+different Pool, and then purging the database records for the first Job.
+
+The Copy process is essentially identical to the Migration feature with the
+exception that the Job that is copied is left unchanged. This essentially
+creates two identical copies of the same backup. However, the copy is treated
+as a copy rather than a backup job, and hence is not directly available for
+restore. If bacula founds a copy when a job record is purged (deleted) from the
+catalog, it will promote the copy as \textsl{real} backup and will make it
+available for automatic restore.
+
+The Copy and the Migration jobs run without using the File daemon by copying
+the data from the old backup Volume to a different Volume in a different Pool.
+
+The section process for which Job or Jobs are migrated
+can be based on quite a number of different criteria such as:
+\begin{itemize}
+\item a single previous Job
+\item a Volume
+\item a Client
+\item a regular expression matching a Job, Volume, or Client name
+\item the time a Job has been on a Volume
+\item high and low water marks (usage or occupation) of a Pool
+\item Volume size
+\end{itemize}
+
+The details of these selection criteria will be defined below.
+
+To run a Migration job, you must first define a Job resource very similar
+to a Backup Job but with {\bf Type = Migrate} instead of {\bf Type =
+Backup}. One of the key points to remember is that the Pool that is
+specified for the migration job is the only pool from which jobs will
+be migrated, with one exception noted below. In addition, the Pool to
+which the selected Job or Jobs will be migrated is defined by the {\bf
+Next Pool = ...} in the Pool resource specified for the Migration Job.
+
+Bacula permits Pools to contain Volumes with different Media Types.
+However, when doing migration, this is a very undesirable condition. For
+migration to work properly, you should use Pools containing only Volumes of
+the same Media Type for all migration jobs.
+
+The migration job normally is either manually started or starts
+from a Schedule much like a backup job. It searches
+for a previous backup Job or Jobs that match the parameters you have
+specified in the migration Job resource, primarily a {\bf Selection Type}
+(detailed a bit later). Then for
+each previous backup JobId found, the Migration Job will run a new Job which
+copies the old Job data from the previous Volume to a new Volume in
+the Migration Pool. It is possible that no prior Jobs are found for
+migration, in which case, the Migration job will simply terminate having
+done nothing, but normally at a minimum, three jobs are involved during a
+migration:
+
+\begin{itemize}
+\item The currently running Migration control Job. This is only
+ a control job for starting the migration child jobs.
+\item The previous Backup Job (already run). The File records
+ for this Job are purged if the Migration job successfully
+ terminates. The original data remains on the Volume until
+ it is recycled and rewritten.
+\item A new Migration Backup Job that moves the data from the
+ previous Backup job to the new Volume. If you subsequently
+ do a restore, the data will be read from this Job.
+\end{itemize}
+
+If the Migration control job finds a number of JobIds to migrate (e.g.
+it is asked to migrate one or more Volumes), it will start one new
+migration backup job for each JobId found on the specified Volumes.
+Please note that Migration doesn't scale too well since Migrations are
+done on a Job by Job basis. This if you select a very large volume or
+a number of volumes for migration, you may have a large number of
+Jobs that start. Because each job must read the same Volume, they will
+run consecutively (not simultaneously).
+
+\section{Migration and Copy Job Resource Directives}
+
+The following directives can appear in a Director's Job resource, and they
+are used to define a Migration job.
+
+\begin{description}
+\item [Pool = \lt{}Pool-name\gt{}] The Pool specified in the Migration
+ control Job is not a new directive for the Job resource, but it is
+ particularly important because it determines what Pool will be examined
+ for finding JobIds to migrate. The exception to this is when {\bf
+ Selection Type = SQLQuery}, and although a Pool directive must still be
+ specified, no Pool is used, unless you specifically include it in the
+ SQL query. Note, in any case, the Pool resource defined by the Pool
+ directove must contain a {\bf Next Pool = ...} directive to define the
+ Pool to which the data will be migrated.
+
+\item [Type = Migrate]
+ {\bf Migrate} is a new type that defines the job that is run as being a
+ Migration Job. A Migration Job is a sort of control job and does not have
+ any Files associated with it, and in that sense they are more or less like
+ an Admin job. Migration jobs simply check to see if there is anything to
+ Migrate then possibly start and control new Backup jobs to migrate the data
+ from the specified Pool to another Pool. Note, any original JobId that
+ is migrated will be marked as having been migrated, and the original
+ JobId can nolonger be used for restores; all restores will be done from
+ the new migrated Job.
+
+
+\item [Type = Copy]
+ {\bf Copy} is a new type that defines the job that is run as being a
+ Copy Job. A Copy Job is a sort of control job and does not have
+ any Files associated with it, and in that sense they are more or less like
+ an Admin job. Copy jobs simply check to see if there is anything to
+ Copy then possibly start and control new Backup jobs to copy the data
+ from the specified Pool to another Pool. Note that when a copy is
+ made, the original JobIds are left unchanged. The new copies can not
+ be used for restoration unless you specifically choose them by JobId.
+ If you subsequently delete a JobId that has a copy, the copy will be
+ automatically upgraded to a Backup rather than a Copy, and it will
+ subsequently be used for restoration.
+
+\item [Selection Type = \lt{}Selection-type-keyword\gt{}]
+ The \lt{}Selection-type-keyword\gt{} determines how the migration job
+ will go about selecting what JobIds to migrate. In most cases, it is
+ used in conjunction with a {\bf Selection Pattern} to give you fine
+ control over exactly what JobIds are selected. The possible values
+ for \lt{}Selection-type-keyword\gt{} are:
+ \begin{description}
+ \item [SmallestVolume] This selection keyword selects the volume with the
+ fewest bytes from the Pool to be migrated. The Pool to be migrated
+ is the Pool defined in the Migration Job resource. The migration
+ control job will then start and run one migration backup job for
+ each of the Jobs found on this Volume. The Selection Pattern, if
+ specified, is not used.
+
+ \item [OldestVolume] This selection keyword selects the volume with the
+ oldest last write time in the Pool to be migrated. The Pool to be
+ migrated is the Pool defined in the Migration Job resource. The
+ migration control job will then start and run one migration backup
+ job for each of the Jobs found on this Volume. The Selection
+ Pattern, if specified, is not used.
+
+ \item [Client] The Client selection type, first selects all the Clients
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Client names, giving
+ a filtered Client name list. All jobs that were backed up for those
+ filtered (regexed) Clients will be migrated.
+ The migration control job will then start and run one migration
+ backup job for each of the JobIds found for those filtered Clients.
+
+ \item [Volume] The Volume selection type, first selects all the Volumes
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Volume names, giving
+ a filtered Volume list. All JobIds that were backed up for those
+ filtered (regexed) Volumes will be migrated.
+ The migration control job will then start and run one migration
+ backup job for each of the JobIds found on those filtered Volumes.
+
+ \item [Job] The Job selection type, first selects all the Jobs (as
+ defined on the {\bf Name} directive in a Job resource)
+ that have been backed up in the Pool specified by the Migration
+ Job resource, then it applies the {\bf Selection Pattern} (defined
+ below) as a regular expression to the list of Job names, giving
+ a filtered Job name list. All JobIds that were run for those
+ filtered (regexed) Job names will be migrated. Note, for a given
+ Job named, they can be many jobs (JobIds) that ran.
+ The migration control job will then start and run one migration
+ backup job for each of the Jobs found.
+
+ \item [SQLQuery] The SQLQuery selection type, used the {\bf Selection
+ Pattern} as an SQL query to obtain the JobIds to be migrated.
+ The Selection Pattern must be a valid SELECT SQL statement for your
+ SQL engine, and it must return the JobId as the first field
+ of the SELECT.
+
+ \item [PoolOccupancy] This selection type will cause the Migration job
+ to compute the total size of the specified pool for all Media Types
+ combined. If it exceeds the {\bf Migration High Bytes} defined in
+ the Pool, the Migration job will migrate all JobIds beginning with
+ the oldest Volume in the pool (determined by Last Write time) until
+ the Pool bytes drop below the {\bf Migration Low Bytes} defined in the
+ Pool. This calculation should be consider rather approximative because
+ it is made once by the Migration job before migration is begun, and
+ thus does not take into account additional data written into the Pool
+ during the migration. In addition, the calculation of the total Pool
+ byte size is based on the Volume bytes saved in the Volume (Media)
+database
+ entries. The bytes calculate for Migration is based on the value stored
+ in the Job records of the Jobs to be migrated. These do not include the
+ Storage daemon overhead as is in the total Pool size. As a consequence,
+ normally, the migration will migrate more bytes than strictly necessary.
+
+ \item [PoolTime] The PoolTime selection type will cause the Migration job to
+ look at the time each JobId has been in the Pool since the job ended.
+ All Jobs in the Pool longer than the time specified on {\bf Migration Time}
+ directive in the Pool resource will be migrated.
+
+ \item [PoolUncopiedJobs] This selection which copies all jobs from a pool
+ to an other pool which were not copied before is available only for copy Jobs.
+
+ \end{description}
+
+\item [Selection Pattern = \lt{}Quoted-string\gt{}]
+ The Selection Patterns permitted for each Selection-type-keyword are
+ described above.
+
+ For the OldestVolume and SmallestVolume, this
+ Selection pattern is not used (ignored).
+
+ For the Client, Volume, and Job
+ keywords, this pattern must be a valid regular expression that will filter
+ the appropriate item names found in the Pool.
+
+ For the SQLQuery keyword, this pattern must be a valid SELECT SQL statement
+ that returns JobIds.
+
+\end{description}
+
+\section{Migration Pool Resource Directives}
+
+The following directives can appear in a Director's Pool resource, and they
+are used to define a Migration job.
+
+\begin{description}
+\item [Migration Time = \lt{}time-specification\gt{}]
+ If a PoolTime migration is done, the time specified here in seconds (time
+ modifiers are permitted -- e.g. hours, ...) will be used. If the
+ previous Backup Job or Jobs selected have been in the Pool longer than
+ the specified PoolTime, then they will be migrated.
+
+\item [Migration High Bytes = \lt{}byte-specification\gt{}]
+ This directive specifies the number of bytes in the Pool which will
+ trigger a migration if a {\bf PoolOccupancy} migration selection
+ type has been specified. The fact that the Pool
+ usage goes above this level does not automatically trigger a migration
+ job. However, if a migration job runs and has the PoolOccupancy selection
+ type set, the Migration High Bytes will be applied. Bacula does not
+ currently restrict a pool to have only a single Media Type, so you
+ must keep in mind that if you mix Media Types in a Pool, the results
+ may not be what you want, as the Pool count of all bytes will be
+ for all Media Types combined.
+
+\item [Migration Low Bytes = \lt{}byte-specification\gt{}]
+ This directive specifies the number of bytes in the Pool which will
+ stop a migration if a {\bf PoolOccupancy} migration selection
+ type has been specified and triggered by more than Migration High
+ Bytes being in the pool. In other words, once a migration job
+ is started with {\bf PoolOccupancy} migration selection and it
+ determines that there are more than Migration High Bytes, the
+ migration job will continue to run jobs until the number of
+ bytes in the Pool drop to or below Migration Low Bytes.
+
+\item [Next Pool = \lt{}pool-specification\gt{}]
+ The Next Pool directive specifies the pool to which Jobs will be
+ migrated. This directive is required to define the Pool into which
+ the data will be migrated. Without this directive, the migration job
+ will terminate in error.
+
+\item [Storage = \lt{}storage-specification\gt{}]
+ The Storage directive specifies what Storage resource will be used
+ for all Jobs that use this Pool. It takes precedence over any other
+ Storage specifications that may have been given such as in the
+ Schedule Run directive, or in the Job resource. We highly recommend
+ that you define the Storage resource to be used in the Pool rather
+ than elsewhere (job, schedule run, ...).
+\end{description}
+
+\section{Important Migration Considerations}
+\index[general]{Important Migration Considerations}
+\begin{itemize}
+\item Each Pool into which you migrate Jobs or Volumes {\bf must}
+ contain Volumes of only one Media Type.
+
+\item Migration takes place on a JobId by JobId basis. That is
+ each JobId is migrated in its entirety and independently
+ of other JobIds. Once the Job is migrated, it will be
+ on the new medium in the new Pool, but for the most part,
+ aside from having a new JobId, it will appear with all the
+ same characteristics of the original job (start, end time, ...).
+ The column RealEndTime in the catalog Job table will contain the
+ time and date that the Migration terminated, and by comparing
+ it with the EndTime column you can tell whether or not the
+ job was migrated. The original job is purged of its File
+ records, and its Type field is changed from "B" to "M" to
+ indicate that the job was migrated.
+
+\item Jobs on Volumes will be Migration only if the Volume is
+ marked, Full, Used, or Error. Volumes that are still
+ marked Append will not be considered for migration. This
+ prevents Bacula from attempting to read the Volume at
+ the same time it is writing it. It also reduces other deadlock
+ situations, as well as avoids the problem that you migrate a
+ Volume and later find new files appended to that Volume.
+
+\item As noted above, for the Migration High Bytes, the calculation
+ of the bytes to migrate is somewhat approximate.
+
+\item If you keep Volumes of different Media Types in the same Pool,
+ it is not clear how well migration will work. We recommend only
+ one Media Type per pool.
+
+\item It is possible to get into a resource deadlock where Bacula does
+ not find enough drives to simultaneously read and write all the
+ Volumes needed to do Migrations. For the moment, you must take
+ care as all the resource deadlock algorithms are not yet implemented.
+
+\item Migration is done only when you run a Migration job. If you set a
+ Migration High Bytes and that number of bytes is exceeded in the Pool
+ no migration job will automatically start. You must schedule the
+ migration jobs, and they must run for any migration to take place.
+
+\item If you migrate a number of Volumes, a very large number of Migration
+ jobs may start.
+
+\item Figuring out what jobs will actually be migrated can be a bit complicated
+ due to the flexibility provided by the regex patterns and the number of
+ different options. Turning on a debug level of 100 or more will provide
+ a limited amount of debug information about the migration selection
+ process.
+
+\item Bacula currently does only minimal Storage conflict resolution, so you
+ must take care to ensure that you don't try to read and write to the
+ same device or Bacula may block waiting to reserve a drive that it
+ will never find. In general, ensure that all your migration
+ pools contain only one Media Type, and that you always
+ migrate to pools with different Media Types.
+
+\item The {\bf Next Pool = ...} directive must be defined in the Pool
+ referenced in the Migration Job to define the Pool into which the
+ data will be migrated.
+
+\item Pay particular attention to the fact that data is migrated on a Job
+ by Job basis, and for any particular Volume, only one Job can read
+ that Volume at a time (no simultaneous read), so migration jobs that
+ all reference the same Volume will run sequentially. This can be a
+ potential bottle neck and does not scale very well to large numbers
+ of jobs.
+
+\item Only migration of Selection Types of Job and Volume have
+ been carefully tested. All the other migration methods (time,
+ occupancy, smallest, oldest, ...) need additional testing.
+
+\item Migration is only implemented for a single Storage daemon. You
+ cannot read on one Storage daemon and write on another.
+\end{itemize}
+
+
+\section{Example Migration Jobs}
+\index[general]{Example Migration Jobs}
+
+When you specify a Migration Job, you must specify all the standard
+directives as for a Job. However, certain such as the Level, Client, and
+FileSet, though they must be defined, are ignored by the Migration job
+because the values from the original job used instead.
+
+As an example, suppose you have the following Job that
+you run every night. To note: there is no Storage directive in the
+Job resource; there is a Storage directive in each of the Pool
+resources; the Pool to be migrated (File) contains a Next Pool
+directive that defines the output Pool (where the data is written
+by the migration job).
+
+\footnotesize
+\begin{verbatim}
+# Define the backup Job
+Job {
+ Name = "NightlySave"
+ Type = Backup
+ Level = Incremental # default
+ Client=rufus-fd
+ FileSet="Full Set"
+ Schedule = "WeeklyCycle"
+ Messages = Standard
+ Pool = Default
+}
+
+# Default pool definition
+Pool {
+ Name = Default
+ Pool Type = Backup
+ AutoPrune = yes
+ Recycle = yes
+ Next Pool = Tape
+ Storage = File
+ LabelFormat = "File"
+}
+
+# Tape pool definition
+Pool {
+ Name = Tape
+ Pool Type = Backup
+ AutoPrune = yes
+ Recycle = yes
+ Storage = DLTDrive
+}
+
+# Definition of File storage device
+Storage {
+ Name = File
+ Address = rufus
+ Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
+ Device = "File" # same as Device in Storage daemon
+ Media Type = File # same as MediaType in Storage daemon
+}
+
+# Definition of DLT tape storage device
+Storage {
+ Name = DLTDrive
+ Address = rufus
+ Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9"
+ Device = "HP DLT 80" # same as Device in Storage daemon
+ Media Type = DLT8000 # same as MediaType in Storage daemon
+}
+
+\end{verbatim}
+\normalsize
+
+Where we have included only the essential information -- i.e. the
+Director, FileSet, Catalog, Client, Schedule, and Messages resources are
+omitted.
+
+As you can see, by running the NightlySave Job, the data will be backed up
+to File storage using the Default pool to specify the Storage as File.
+
+Now, if we add the following Job resource to this conf file.
+
+\footnotesize
+\begin{verbatim}
+Job {
+ Name = "migrate-volume"
+ Type = Migrate
+ Level = Full
+ Client = rufus-fd
+ FileSet = "Full Set"
+ Messages = Standard
+ Pool = Default
+ Maximum Concurrent Jobs = 4
+ Selection Type = Volume
+ Selection Pattern = "File"
+}
+\end{verbatim}
+\normalsize
+
+and then run the job named {\bf migrate-volume}, all volumes in the Pool
+named Default (as specified in the migrate-volume Job that match the
+regular expression pattern {\bf File} will be migrated to tape storage
+DLTDrive because the {\bf Next Pool} in the Default Pool specifies that
+Migrations should go to the pool named {\bf Tape}, which uses
+Storage {\bf DLTDrive}.
+
+If instead, we use a Job resource as follows:
+
+\footnotesize
+\begin{verbatim}
+Job {
+ Name = "migrate"
+ Type = Migrate
+ Level = Full
+ Client = rufus-fd
+ FileSet="Full Set"
+ Messages = Standard
+ Pool = Default
+ Maximum Concurrent Jobs = 4
+ Selection Type = Job
+ Selection Pattern = ".*Save"
+}
+\end{verbatim}
+\normalsize
+
+All jobs ending with the name Save will be migrated from the File Default to
+the Tape Pool, or from File storage to Tape storage.
projects / bacula / docs / blobdiff