Most people prefer to have a Pool of tapes that are used for daily backups and
recycled once a week, another Pool of tapes that are used for Full backups
once a week and recycled monthly, and finally a Pool of tapes that are used
-once a month and recycled after a year or two. With a scheme like this, the
+once a month and recycled after a year or two. With a scheme like this, the
number of tapes in your pool or pools remains constant.
By properly defining your Volume Pools with appropriate Retention periods,
-Bacula can manage the recycling (such as defined above) automatically.
+Bacula can manage the recycling (such as defined above) automatically.
Automatic recycling of Volumes is controlled by four records in the {\bf
Pool} resource definition in the Director's configuration file. These four
-records are:
+records are:
-\begin{itemize}
-\item AutoPrune = yes
-\item VolumeRetention = \lt{}time\gt{}
+\begin{bsysitemize}
+\item AutoPrune = yes
+\item VolumeRetention = \lt{}time\gt{}
\item Recycle = yes
-\item RecyclePool = \lt{}APool\gt{} (\textit{This require bacula 2.1.4 or greater})
-\end{itemize}
+\item RecyclePool = \lt{}APool\gt{} %(\textit{This require bacula 2.1.4 or greater})
+\end{bsysitemize}
The above three directives are all you need assuming that you fill
-each of your Volumes then wait the Volume Retention period before
+each of your Volumes then wait the Volume Retention period before
reusing them. If you want Bacula to stop using a Volume and recycle
-it before it is full, you will need to use one or more additional
+it before it is full, you will need to use one or more additional
directives such as:
-\begin{itemize}
+\begin{bsysitemize}
\item Use Volume Once = yes
\item Volume Use Duration = ttt
\item Maximum Volume Jobs = nnn
\item Maximum Volume Bytes = mmm
-\end{itemize}
-Please see below and
+\end{bsysitemize}
+Please see below and
the \ilink{Basic Volume Management}{DiskChapter} chapter
-of this manual for more complete examples.
+of this manual for more complete examples.
Automatic recycling of Volumes is performed by Bacula only when it wants a
new Volume and no appendable Volumes are available in the Pool. It will then
-search the Pool for any Volumes with the {\bf Recycle} flag set and the
-Volume Status is {\bf Purged}. At that point, it will choose the oldest
+search the Pool for any Volumes with the {\bf Recycle} flag set and the
+Volume Status is {\bf Purged}. At that point, it will choose the oldest
purged volume and recycle it.
-If there are no volumes with Status {\bf Purged}, then
+If there are no volumes with Status {\bf Purged}, then
the recycling occurs in two steps:
The first is that the Catalog for a Volume must be pruned of all Jobs (i.e.
Purged). Files contained on that Volume, and the second step is the actual
something. However, the catalog will grow larger and larger as time goes
on, and eventually it can become unacceptably large.
-Bacula's process for removing entries from the catalog is called Pruning.
-The default is Automatic Pruning, which means that once an entry reaches a
-certain age (e.g. 30 days old) it is removed from the catalog. Once a job
-has been pruned, you can still restore it from the backup tape, but one
-additional step is required: scanning the volume with bscan. The
-alternative to Automatic Pruning is Manual Pruning, in which you explicitly
-tell Bacula to erase the catalog entries for a volume. You'd usually do
-this when you want to reuse a Bacula volume, because there's no point in
-keeping a list of files that USED TO BE on a tape. Or, if the catalog is
-starting to get too big, you could prune the oldest jobs to save space.
-Manual pruning is done with the \ilink{ prune command}{ManualPruning} in
-the console. (thanks to Bryce Denney for the above explanation).
+Bacula's process for removing entries from the catalog is called Pruning. The
+default is Automatic Pruning, which means that once an entry reaches a certain
+age (e.g. 30 days old) it is removed from the catalog. Note that Job records
+that are required for current restore won't be removed automatically, and File
+records are needed for VirtualFull and Accurate backups. Once a job has been
+pruned, you can still restore it from the backup tape, but one additional step
+is required: scanning the volume with bscan. The alternative to Automatic
+Pruning is Manual Pruning, in which you explicitly tell Bacula to erase the
+catalog entries for a volume. You'd usually do this when you want to reuse a
+Bacula volume, because there's no point in keeping a list of files that USED TO
+BE on a tape. Or, if the catalog is starting to get too big, you could prune
+the oldest jobs to save space. Manual pruning is done with the
+\bsysxrlink{prune}{ManualPruning}{console}{command} in the \consoleman{}
+ (thanks to Bryce Denney for the above explanation).
\section{Pruning Directives}
\index[general]{Pruning Directives }
The durations inter-depend a bit because if Bacula prunes a Volume, it
automatically removes all the Job records, and all the File records. Also when
a Job record is pruned, all the File records for that Job are also pruned
-(deleted) from the catalog.
+(deleted) from the catalog.
Having the File records in the database means that you can examine all the
files backed up for a particular Job. They take the most space in the catalog
(probably 90-95\% of the total). When the File records are pruned, the Job
records can remain, and you can still examine what Jobs ran, but not the
details of the Files backed up. In addition, without the File records, you
-cannot use the Console restore command to restore the files.
+cannot use the Console restore command to restore the files.
When a Job record is pruned, the Volume (Media record) for that Job can still
remain in the database, and if you do a "list volumes", you will see the
volume information, but the Job records (and its File records) will no longer
-be available.
+be available.
In each case, pruning removes information about where older files are, but it
also prevents the catalog from growing to be too large. You choose the
time periods you want to keep those records online, and the size of the
database. You can always re-insert the records (with 98\% of the original data)
by using "bscan" to scan in a whole Volume or any part of the volume that
-you want.
+you want.
By setting {\bf AutoPrune} to {\bf yes} you will permit {\bf Bacula} to
automatically prune all Volumes in the Pool when a Job needs another Volume.
change to the physical data on the Volume occurs during the pruning process.
When all files are pruned from a Volume (i.e. no records in the catalog), the
Volume will be marked as {\bf Purged} implying that no Jobs remain on the
-volume. The Pool records that control the pruning are described below.
+volume. The Pool records that control the pruning are described below.
\begin{description}
Bacula will prune all Volumes that can be pruned (i.e. AutoPrune set) in an
attempt to find a usable volume. If during the autoprune, all files are
pruned from the Volume, it will be marked with VolStatus {\bf Purged}. The
- default is {\bf yes}. Note, that although the File and Job records may be
- pruned from the catalog, a Volume will be marked Purged (and hence
+ default is {\bf yes}. Note, that although the File and Job records may be
+ pruned from the catalog, a Volume will be marked Purged (and hence
ready for recycling) if the Volume status is Append, Full, Used, or Error.
If the Volume has another status, such as Archive, Read-Only, Disabled,
Busy, or Cleaning, the Volume status will not be changed to Purged.
When all file catalog entries are removed from the volume, its VolStatus is
set to {\bf Purged}. The files remain physically on the Volume until the
- volume is overwritten.
+ volume is overwritten.
Retention periods are specified in seconds, minutes, hours, days, weeks,
- months, quarters, or years on the record. See the
+ months, quarters, or years on the record. See the
\ilink{Configuration chapter}{Time} of this manual for
- additional details of time specification.
+ additional details of time specification.
-The default is 1 year.
+The default is 1 year.
% TODO: if that is the format, should it be in quotes? decide on a style
\item [Recycle = \lt{}yes|no\gt{}]
\end{description}
It is also possible to "force" pruning of all Volumes in the Pool
- associated with a Job by adding {\bf Prune Files = yes} to the Job resource.
+ associated with a Job by adding {\bf Prune Files = yes} to the Job resource.
\label{Recycling}
\label{RecyclingAlgorithm}
when a Job needs a new Volume and no appendable Volumes are available), Bacula
will look for the oldest Volume that is Purged (all Jobs and Files expired),
and if the {\bf Recycle} flag is on (Recycle=yes) for that Volume, Bacula will
-relabel it and write new data on it.
+relabel it and write new data on it.
As mentioned above, there are two key points for getting a Volume
to be recycled. First, the Volume must no longer be marked Append (there
that reference that Volume. Once both those conditions are satisfied,
the volume can be marked Purged and hence recycled.
-The full algorithm that Bacula uses when it needs a new Volume is:
+The full algorithm that Bacula uses when it needs a new Volume is:
\index[general]{New Volume Algorithm}
\index[general]{Algorithm!New Volume}
The algorithm described below assumes that AutoPrune is enabled,
that Recycling is turned on, and that you have defined
appropriate Retention periods, or used the defaults for all these
-items.
+items.
-\begin{itemize}
+\begin{bsysitemize}
\item If the request is for an Autochanger device, look only
for Volumes in the Autochanger (i.e. with InChanger set and that have
- the correct Storage device).
+ the correct Storage device).
\item Search the Pool for a Volume with VolStatus=Append (if there is more
than one, the Volume with the oldest date last written is chosen. If
two have the same date then the one with the lowest MediaId is chosen).
date last written is chosen. If two have the same date then the one
with the lowest MediaId is chosen).
\item Try recycling any purged Volumes.
-\item Prune volumes applying Volume retention period (Volumes with VolStatus
+\item Prune volumes applying Volume retention period (Volumes with VolStatus
Full, Used, or Append are pruned). Note, even if all the File and Job
records are pruned from a Volume, the Volume will not be marked Purged
until the Volume retention period expires.
-\item Search the Pool for a Volume with VolStatus=Purged
+\item Search the Pool for a Volume with VolStatus=Purged
\item If a Pool named "Scratch" exists, search for a Volume and if found
move it to the current Pool for the Job and use it. Note, when
the Scratch Volume is moved into the current Pool, the basic
(equivalent to an {\bf update volume from pool} command).
\item If we were looking for Volumes in the Autochanger, go back to
step 2 above, but this time, look for any Volume whether or not
- it is in the Autochanger.
-\item Attempt to create a new Volume if automatic labeling enabled
+ it is in the Autochanger.
+\item Attempt to create a new Volume if automatic labeling enabled
If Python is enabled, a Python NewVolume event is generated before
the Label Format directve is used. If the maximum number of Volumes
specified for the pool is reached, a new Volume will not be created.
\item Prune the oldest Volume if RecycleOldestVolume=yes (the Volume with the
oldest LastWritten date and VolStatus equal to Full, Recycle, Purged, Used,
or Append is chosen). This record ensures that all retention periods are
- properly respected.
+ properly respected.
\item Purge the oldest Volume if PurgeOldestVolume=yes (the Volume with the
oldest LastWritten date and VolStatus equal to Full, Recycle, Purged, Used,
or Append is chosen). We strongly recommend against the use of {\bf
PurgeOldestVolume} as it can quite easily lead to loss of current backup
- data.
-\item Give up and ask operator.
-\end{itemize}
+ data.
+\item Give up and ask operator.
+\end{bsysitemize}
The above occurs when Bacula has finished writing a Volume or when no Volume
-is present in the drive.
+is present in the drive.
On the other hand, if you have inserted a different Volume after the last job,
and Bacula recognizes the Volume as valid, it will request authorization from
the Director to use this Volume. In this case, if you have set {\bf Recycle
Current Volume = yes} and the Volume is marked as Used or Full, Bacula will
prune the volume and if all jobs were removed during the pruning (respecting
-the retention periods), the Volume will be recycled and used.
+the retention periods), the Volume will be recycled and used.
-The recycling algorithm in this case is:
-\begin{itemize}
-\item If the VolStatus is {\bf Append} or {\bf Recycle}
- is set, the volume will be used.
+The recycling algorithm in this case is:
+\begin{bsysitemize}
+\item If the VolStatus is {\bf Append} or {\bf Recycle}
+ is set, the volume will be used.
\item If {\bf Recycle Current Volume} is set and the volume is marked {\bf
Full} or {\bf Used}, Bacula will prune the volume (applying the retention
- period). If all Jobs are pruned from the volume, it will be recycled.
-\end{itemize}
+ period). If all Jobs are pruned from the volume, it will be recycled.
+\end{bsysitemize}
This permits users to manually change the Volume every day and load tapes in
an order different from what is in the catalog, and if the volume does not
-contain a current copy of your backup data, it will be used.
+contain a current copy of your backup data, it will be used.
A few points from Alan Brown to keep in mind:
\begin{enumerate}
-\item If a pool doesn't have maximum volumes defined then Bacula will prefer to
+\item If a pool doesn't have maximum volumes defined then Bacula will prefer to
demand new volumes over forcibly purging older volumes.
\item If volumes become free through pruning and the Volume retention period has
record when the Media record is created (normally when the Volume is labeled).
This Recycle status is stored in the Media record of the Catalog. Using
the Console program, you may subsequently change the Recycle status for each
-Volume. For example in the following output from {\bf list volumes}:
+Volume. For example in the following output from {\bf list volumes}:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
+----------+-------+--------+---------+------------+--------+-----+
| VolumeNa | Media | VolSta | VolByte | LastWritte | VolRet | Rec |
+----------+-------+--------+---------+------------+--------+-----+
| File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
| File0007 | File | Purged | 1896466 | 2002-05-26 | 14400 | 1 |
+----------+-------+--------+---------+------------+--------+-----+
-\end{verbatim}
+\end{lstlisting}
\normalsize
all the volumes are marked as recyclable, and the last Volume, {\bf File0007}
is still recoverable. A purged Volume simply means that there are no entries
in the Catalog. Even if the Volume Status is changed to {\bf Recycle}, the
data on the Volume will be recoverable. The data is lost only when the Volume
-is re-labeled and re-written.
+is re-labeled and re-written.
To modify Volume {\bf File0001} so that it cannot be recycled, you use the
{\bf update volume pool=File} command in the console program, or simply {\bf
-update} and Bacula will prompt you for the information.
+update} and Bacula will prompt you for the information.
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
+----------+------+-------+---------+-------------+-------+-----+
| VolumeNa | Media| VolSta| VolByte | LastWritten | VolRet| Rec |
+----------+------+-------+---------+-------------+-------+-----+
| File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
| File0007 | File | Purged| 1896466 | 2002-05-26 | 14400 | 1 |
+----------+------+-------+---------+-------------+-------+-----+
-\end{verbatim}
+\end{lstlisting}
\normalsize
In this case, {\bf File0001} will never be automatically recycled. The same
-effect can be achieved by setting the Volume Status to Read-Only.
+effect can be achieved by setting the Volume Status to Read-Only.
-As you have noted, the Volume Status (VolStatus) column in the
+As you have noted, the Volume Status (VolStatus) column in the
catalog database contains the current status of the Volume, which
is normally maintained automatically by Bacula. To give you an
idea of some of the values it can take during the life cycle of
a Volume, here is a picture created by Arno Lehmann:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
A typical volume life cycle is like this:
because job count or size limit exceeded
Recycled <-------------------------------------- Purged
Volume is selected for reuse
-\end{verbatim}
+\end{lstlisting}
\normalsize
first time it is written, Bacula will simply append to the tape and eventually
request another volume. Using the tape only once, forces the tape to be marked
{\bf Full} after each use, and the next time {\bf Bacula} runs, it will
-recycle the tape.
+recycle the tape.
-An example Pool resource that does this is:
+An example Pool resource that does this is:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Pool {
Name = DDS-4
Use Volume Once = yes
VolumeRetention = 12h # expire after 12 hours
Recycle = yes
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
\section{Daily, Weekly, Monthly Tape Usage Example}
This example is meant to show you how one could define a fixed set of volumes
that Bacula will rotate through on a regular schedule. There are an infinite
number of such schemes, all of which have various advantages and
-disadvantages.
+disadvantages.
-We start with the following assumptions:
+We start with the following assumptions:
-\begin{itemize}
-\item A single tape has more than enough capacity to do a full save.
+\begin{bsysitemize}
+\item A single tape has more than enough capacity to do a full save.
\item There are ten tapes that are used on a daily basis for incremental
- backups. They are prelabeled Daily1 ... Daily10.
+ backups. They are prelabeled Daily1 ... Daily10.
\item There are four tapes that are used on a weekly basis for full backups.
- They are labeled Week1 ... Week4.
+ They are labeled Week1 ... Week4.
\item There are 12 tapes that are used on a monthly basis for full backups.
- They are numbered Month1 ... Month12
+ They are numbered Month1 ... Month12
\item A full backup is done every Saturday evening (tape inserted Friday
- evening before leaving work).
-\item No backups are done over the weekend (this is easy to change).
+ evening before leaving work).
+\item No backups are done over the weekend (this is easy to change).
\item The first Friday of each month, a Monthly tape is used for the Full
- backup.
+ backup.
\item Incremental backups are done Monday - Friday (actually Tue-Fri
- mornings).
+ mornings).
% TODO: why this "actually"? does this need to be explained?
- \end{itemize}
+ \end{bsysitemize}
We start the system by doing a Full save to one of the weekly volumes or one
of the monthly volumes. The next morning, we remove the tape and insert a
tape in the series rather than a Weekly tape, then continue. When a Daily tape
finally fills up, {\bf Bacula} will request the next one in the series, and
the next day when you notice the email message, you will mount it and {\bf
-Bacula} will finish the unfinished incremental backup.
+Bacula} will finish the unfinished incremental backup.
What does this give? Well, at any point, you will have the last complete
Full save plus several Incremental saves. For any given file you want to
for at least the last 14 days. For older versions, you will have at least three
and probably four Friday full saves of that file, and going back further, you
will have a copy of that file made on the beginning of the month for at least
-a year.
+a year.
So you have copies of any file (or your whole system) for at least a year, but
as you go back in time, the time between copies increases from daily to weekly
-to monthly.
+to monthly.
-What would the Bacula configuration look like to implement such a scheme?
+What would the Bacula configuration look like to implement such a scheme?
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Schedule {
Name = "NightlySave"
Run = Level=Full Pool=Monthly 1st sat at 03:05
}
FileSet {
Name = "File Set"
- Include = signature=MD5 {
- fffffffffffffffff
+ Include {
+ Options { signature=MD5 }
+ File = fffffffffffffffff
}
- Exclude = { *.o }
+ Exclude { File=*.o }
}
Pool {
Name = Daily
VolumeRetention = 365d # recycle in 1 year
Recycle = yes
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
\section{ Automatic Pruning and Recycling Example}
Perhaps the best way to understand the various resource records that come into
play during automatic pruning and recycling is to run a Job that goes through
the whole cycle. If you add the following resources to your Director's
-configuration file:
+configuration file:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Schedule {
Name = "30 minute cycle"
Run = Level=Full Pool=File Messages=Standard Storage=File
Media Type = File
}
FileSet {
- Name = "Test Files"
- Include = signature=MD5 {
- fffffffffffffffff
+ Name = "File Set"
+ Include {
+ Options { signature=MD5 }
+ File = fffffffffffffffff
}
- Exclude = { *.o }
+ Exclude { File=*.o }
}
Pool {
Name = File
Maximum Volumes = 12
Recycle = yes
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
Where you will need to replace the {\bf ffffffffff}'s by the appropriate files
to be saved for your configuration. For the FileSet Include, choose a
directory that has one or two megabytes maximum since there will probably be
-approximately eight copies of the directory that {\bf Bacula} will cycle through.
+approximately eight copies of the directory that {\bf Bacula} will cycle through.
In addition, you will need to add the following to your Storage daemon's
-configuration file:
+configuration file:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
Device {
Name = FileStorage
Media Type = File
RemovableMedia = no;
AlwaysOpen = no;
}
-\end{verbatim}
+\end{lstlisting}
\normalsize
With the above resources, Bacula will start a Job every half hour that saves a
copy of the directory you chose to /tmp/File0001 ... /tmp/File0012. After 4
hours, Bacula will start recycling the backup Volumes (/tmp/File0001 ...). You
should see this happening in the output produced. Bacula will automatically
-create the Volumes (Files) the first time it uses them.
+create the Volumes (Files) the first time it uses them.
To turn it off, either delete all the resources you've added, or simply
-comment out the {\bf Schedule} record in the {\bf Job} resource.
+comment out the {\bf Schedule} record in the {\bf Job} resource.
\section{Manually Recycling Volumes}
\label{manualrecycling}
\index[general]{Manually Recycling Volumes }
Although automatic recycling of Volumes is implemented in version 1.20 and
-later (see the
+later (see the
\ilink{Automatic Recycling of Volumes}{RecyclingChapter} chapter of
-this manual), you may want to manually force reuse (recycling) of a Volume.
+this manual), you may want to manually force reuse (recycling) of a Volume.
Assuming that you want to keep the Volume name, but you simply want to write
-new data on the tape, the steps to take are:
+new data on the tape, the steps to take are:
-\begin{itemize}
+\begin{bsysitemize}
\item Use the {\bf update volume} command in the Console to ensure that the
- {\bf Recycle} field is set to {\bf 1}
+ {\bf Recycle} field is set to {\bf 1}
\item Use the {\bf purge jobs volume} command in the Console to mark the
- Volume as {\bf Purged}. Check by using {\bf list volumes}.
-\end{itemize}
+ Volume as {\bf Purged}. Check by using {\bf list volumes}.
+\end{bsysitemize}
Once the Volume is marked Purged, it will be recycled the next time a Volume
-is needed.
+is needed.
If you wish to reuse the tape by giving it a new name, follow the following
-steps:
+steps:
-\begin{itemize}
+\begin{bsysitemize}
\item Use the {\bf purge jobs volume} command in the Console to mark the
- Volume as {\bf Purged}. Check by using {\bf list volumes}.
+ Volume as {\bf Purged}. Check by using {\bf list volumes}.
\item In Bacula version 1.30 or greater, use the Console {\bf relabel}
- command to relabel the Volume.
-\end{itemize}
+ command to relabel the Volume.
+\end{bsysitemize}
-Please note that the relabel command applies only to tape Volumes.
+Please note that the relabel command applies only to tape Volumes.
For Bacula versions prior to 1.30 or to manually relabel the Volume, use the
-instructions below:
+instructions below:
-\begin{itemize}
+\begin{bsysitemize}
\item Use the {\bf delete volume} command in the Console to delete the Volume
- from the Catalog.
+ from the Catalog.
\item If a different tape is mounted, use the {\bf unmount} command,
- remove the tape, and insert the tape to be renamed.
-\item Write an EOF mark in the tape using the following commands:
+ remove the tape, and insert the tape to be renamed.
+\item Write an EOF mark in the tape using the following commands:
\footnotesize
-\begin{verbatim}
+\begin{lstlisting}
mt -f /dev/nst0 rewind
mt -f /dev/nst0 weof
-\end{verbatim}
+\end{lstlisting}
\normalsize
where you replace {\bf /dev/nst0} with the appropriate device name on your
-system.
+system.
\item Use the {\bf label} command to write a new label to the tape and to
- enter it in the catalog.
-\end{itemize}
+ enter it in the catalog.
+\end{bsysitemize}
Please be aware that the {\bf delete} command can be dangerous. Once it is
done, to recover the File records, you must either restore your database as it
was before the {\bf delete} command, or use the {\bf bscan} utility program to
-scan the tape and recreate the database entries.
+scan the tape and recreate the database entries.
projects / bacula / docs / blobdiff