--- /dev/null
+%%
+%%
+
+\chapter{Automated Disk Backup}
+\label{PoolsChapter}
+\index[general]{Volumes!Using Pools to Manage}
+\index[general]{Disk!Automated Backup}
+\index[general]{Using Pools to Manage Volumes}
+\index[general]{Automated Disk Backup}
+
+If you manage five or ten machines and have a nice tape backup, you don't need
+Pools, and you may wonder what they are good for. In this chapter, you will
+see that Pools can help you optimize disk storage space. The same techniques
+can be applied to a shop that has multiple tape drives, or that wants to mount
+various different Volumes to meet their needs.
+
+The rest of this chapter will give an example involving backup to disk
+Volumes, but most of the information applies equally well to tape Volumes.
+
+\label{TheProblem}
+\section{The Problem}
+\index[general]{Problem}
+
+A site that I administer (a charitable organization) had a tape DDS-3 tape
+drive that was failing. The exact reason for the failure is still unknown.
+Worse yet, their full backup size is about 15GB whereas the capacity of their
+broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the
+necessary cassettes was more expensive than their budget could handle.
+
+\label{TheSolution}
+\section{The Solution}
+\index[general]{Solution}
+
+They want to maintain six months of backup data, and be able to access the old
+files on a daily basis for a week, a weekly basis for a month, then monthly
+for six months. In addition, offsite capability was not needed (well perhaps
+it really is, but it was never used). Their daily changes amount to about
+300MB on the average, or about 2GB per week.
+
+As a consequence, the total volume of data they need to keep to meet their
+needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
+
+The chosen solution was to buy a 120GB hard disk for next to nothing -- far
+less than 1/10th the price of a tape drive and the cassettes to handle the
+same amount of data, and to have Bacula write to disk files.
+
+The rest of this chapter will explain how to setup Bacula so that it would
+automatically manage a set of disk files with the minimum sysadmin
+intervention. The system has been running since 22 January 2004 until today
+(23 June 2007) with no intervention, with the exception of adding
+a second 120GB hard disk after a year because their needs grew
+over that time to more than the 120GB (168GB to be exact). The only other
+intervention I have made is a periodic (about once a year) Bacula upgrade.
+
+\label{OverallDesign}
+\section{Overall Design}
+\index[general]{Overall Design}
+\index[general]{Design!Overall}
+
+Getting Bacula to write to disk rather than tape in the simplest case is
+rather easy, and is documented in the previous chapter. In addition, all the
+directives discussed here are explained in that chapter. We'll leave it to you
+to look at the details there. If you haven't read it and are not familiar with
+Pools, you probably should at least read it once quickly for the ideas before
+continuing here.
+
+One needs to consider about what happens if we have only a single large Bacula
+Volume defined on our hard disk. Everything works fine until the Volume fills,
+then Bacula will ask you to mount a new Volume. This same problem applies to
+the use of tape Volumes if your tape fills. Being a hard disk and the only one
+you have, this will be a bit of a problem. It should be obvious that it is
+better to use a number of smaller Volumes and arrange for Bacula to
+automatically recycle them so that the disk storage space can be reused. The
+other problem with a single Volume, is that until version 2.0.0,
+Bacula did not seek within a disk Volume, so restoring a single file can take
+more time than one would expect.
+
+As mentioned, the solution is to have multiple Volumes, or files on the disk.
+To do so, we need to limit the use and thus the size of a single Volume, by
+time, by number of jobs, or by size. Any of these would work, but we chose to
+limit the use of a single Volume by putting a single job in each Volume with
+the exception of Volumes containing Incremental backup where there will be 6
+jobs (a week's worth of data) per volume. The details of this will be
+discussed shortly. This is a single client backup, so if you have multiple
+clients you will need to multiply those numbers by the number of clients,
+or use a different system for switching volumes, such as limiting the
+volume size.
+
+The next problem to resolve is recycling of Volumes. As you noted from above,
+the requirements are to be able to restore monthly for 6 months, weekly for a
+month, and daily for a week. So to simplify things, why not do a Full save
+once a month, a Differential save once a week, and Incremental saves daily.
+Now since each of these different kinds of saves needs to remain valid for
+differing periods, the simplest way to do this (and possibly the only) is to
+have a separate Pool for each backup type.
+
+The decision was to use three Pools: one for Full saves, one for Differential
+saves, and one for Incremental saves, and each would have a different number
+of volumes and a different Retention period to accomplish the requirements.
+
+\label{FullPool}
+\subsection{Full Pool}
+\index[general]{Pool!Full}
+\index[general]{Full Pool}
+
+Putting a single Full backup on each Volume, will require six Full save
+Volumes, and a retention period of six months. The Pool needed to do that is:
+
+\footnotesize
+\begin{verbatim}
+Pool {
+ Name = Full-Pool
+ Pool Type = Backup
+ Recycle = yes
+ AutoPrune = yes
+ Volume Retention = 6 months
+ Maximum Volume Jobs = 1
+ Label Format = Full-
+ Maximum Volumes = 9
+}
+\end{verbatim}
+\normalsize
+
+Since these are disk Volumes, no space is lost by having separate Volumes for
+each backup (done once a month in this case). The items to note are the
+retention period of six months (i.e. they are recycled after six months), that
+there is one job per volume (Maximum Volume Jobs = 1), the volumes will be
+labeled Full-0001, ... Full-0006 automatically. One could have labeled these
+manually from the start, but why not use the features of Bacula.
+
+Six months after the first volume is used, it will be subject to pruning
+and thus recycling, so with a maximum of 9 volumes, there should always be
+3 volumes available (note, they may all be marked used, but they will be
+marked purged and recycled as needed).
+
+If you have two clients, you would want to set {\bf Maximum Volume Jobs} to
+2 instead of one, or set a limit on the size of the Volumes, and possibly
+increase the maximum number of Volumes.
+
+
+\label{DiffPool}
+\subsection{Differential Pool}
+\index[general]{Pool!Differential}
+\index[general]{Differential Pool}
+
+For the Differential backup Pool, we choose a retention period of a bit longer
+than a month and ensure that there is at least one Volume for each of the
+maximum of five weeks in a month. So the following works:
+
+\footnotesize
+\begin{verbatim}
+Pool {
+ Name = Diff-Pool
+ Pool Type = Backup
+ Recycle = yes
+ AutoPrune = yes
+ Volume Retention = 40 days
+ Maximum Volume Jobs = 1
+ Label Format = Diff-
+ Maximum Volumes = 10
+}
+\end{verbatim}
+\normalsize
+
+As you can see, the Differential Pool can grow to a maximum of 9 volumes,
+and the Volumes are retained 40 days and thereafter they can be recycled. Finally
+there is one job per volume. This, of course, could be tightened up a lot, but
+the expense here is a few GB which is not too serious.
+
+If a new volume is used every week, after 40 days, one will have used 7
+volumes, and there should then always be 3 volumes that can be purged and
+recycled.
+
+See the discussion above concering the Full pool for how to handle multiple
+clients.
+
+\label{IncPool}
+\subsection{Incremental Pool}
+\index[general]{Incremental Pool}
+\index[general]{Pool!Incremental}
+
+Finally, here is the resource for the Incremental Pool:
+
+\footnotesize
+\begin{verbatim}
+Pool {
+ Name = Inc-Pool
+ Pool Type = Backup
+ Recycle = yes
+ AutoPrune = yes
+ Volume Retention = 20 days
+ Maximum Volume Jobs = 6
+ Label Format = Inc-
+ Maximum Volumes = 7
+}
+\end{verbatim}
+\normalsize
+
+We keep the data for 20 days rather than just a week as the needs require. To
+reduce the proliferation of volume names, we keep a week's worth of data (6
+incremental backups) in each Volume. In practice, the retention period should
+be set to just a bit more than a week and keep only two or three volumes
+instead of five. Again, the lost is very little and as the system reaches the
+full steady state, we can adjust these values so that the total disk usage
+doesn't exceed the disk capacity.
+
+If you have two clients, the simplest thing to do is to increase the
+maximum volume jobs from 6 to 12. As mentioned above, it is also possible
+limit the size of the volumes. However, in that case, you will need to
+have a better idea of the volume or add sufficient volumes to the pool so
+that you will be assured that in the next cycle (after 20 days) there is
+at least one volume that is pruned and can be recycled.
+
+
+\label{Example}
+\section{The Actual Conf Files}
+\index[general]{Files!Actual Conf}
+\index[general]{Actual Conf Files}
+
+The following example shows you the actual files used, with only a few minor
+modifications to simplify things.
+
+The Director's configuration file is as follows:
+
+\footnotesize
+\begin{verbatim}
+Director { # define myself
+ Name = bacula-dir
+ DIRport = 9101
+ QueryFile = "/home/bacula/bin/query.sql"
+ WorkingDirectory = "/home/bacula/working"
+ PidDirectory = "/home/bacula/working"
+ Maximum Concurrent Jobs = 1
+ Password = " *** CHANGE ME ***"
+ Messages = Standard
+}
+# By default, this job will back up to disk in /tmp
+Job {
+ Name = client
+ Type = Backup
+ Client = client-fd
+ FileSet = "Full Set"
+ Schedule = "WeeklyCycle"
+ Storage = File
+ Messages = Standard
+ Pool = Default
+ Full Backup Pool = Full-Pool
+ Incremental Backup Pool = Inc-Pool
+ Differential Backup Pool = Diff-Pool
+ Write Bootstrap = "/home/bacula/working/client.bsr"
+ Priority = 10
+}
+
+# Backup the catalog database (after the nightly save)
+Job {
+ Name = "BackupCatalog"
+ Type = Backup
+ Client = client-fd
+ FileSet="Catalog"
+ Schedule = "WeeklyCycleAfterBackup"
+ Storage = File
+ Messages = Standard
+ Pool = Default
+ # This creates an ASCII copy of the catalog
+ # WARNING!!! Passing the password via the command line is insecure.
+ # see comments in make_catalog_backup for details.
+ RunBeforeJob = "/home/bacula/bin/make_catalog_backup bacula bacula"
+ # This deletes the copy of the catalog
+ RunAfterJob = "/home/bacula/bin/delete_catalog_backup"
+ Write Bootstrap = "/home/bacula/working/BackupCatalog.bsr"
+ Priority = 11 # run after main backup
+}
+
+# Standard Restore template, to be changed by Console program
+Job {
+ Name = "RestoreFiles"
+ Type = Restore
+ Client = havana-fd
+ FileSet="Full Set"
+ Storage = File
+ Messages = Standard
+ Pool = Default
+ Where = /tmp/bacula-restores
+}
+
+
+
+# List of files to be backed up
+FileSet {
+ Name = "Full Set"
+ Include = { Options { signature=SHA1; compression=GZIP9 }
+ File = /
+ File = /usr
+ File = /home
+ File = /boot
+ File = /var
+ File = /opt
+ }
+ Exclude = {
+ File = /proc
+ File = /tmp
+ File = /.journal
+ File = /.fsck
+ ...
+ }
+}
+Schedule {
+ Name = "WeeklyCycle"
+ Run = Level=Full 1st sun at 2:05
+ Run = Level=Differential 2nd-5th sun at 2:05
+ Run = Level=Incremental mon-sat at 2:05
+}
+
+# This schedule does the catalog. It starts after the WeeklyCycle
+Schedule {
+ Name = "WeeklyCycleAfterBackup"
+ Run = Level=Full sun-sat at 2:10
+}
+
+# This is the backup of the catalog
+FileSet {
+ Name = "Catalog"
+ Include { Options { signature=MD5 }
+ File = /home/bacula/working/bacula.sql
+ }
+}
+
+Client {
+ Name = client-fd
+ Address = client
+ FDPort = 9102
+ Catalog = MyCatalog
+ Password = " *** CHANGE ME ***"
+ AutoPrune = yes # Prune expired Jobs/Files
+ Job Retention = 6 months
+ File Retention = 60 days
+}
+
+Storage {
+ Name = File
+ Address = localhost
+ SDPort = 9103
+ Password = " *** CHANGE ME ***"
+ Device = FileStorage
+ Media Type = File
+}
+
+Catalog {
+ Name = MyCatalog
+ dbname = bacula; user = bacula; password = ""
+}
+
+Pool {
+ Name = Full-Pool
+ Pool Type = Backup
+ Recycle = yes # automatically recycle Volumes
+ AutoPrune = yes # Prune expired volumes
+ Volume Retention = 6 months
+ Maximum Volume Jobs = 1
+ Label Format = Full-
+ Maximum Volumes = 9
+}
+
+Pool {
+ Name = Inc-Pool
+ Pool Type = Backup
+ Recycle = yes # automatically recycle Volumes
+ AutoPrune = yes # Prune expired volumes
+ Volume Retention = 20 days
+ Maximum Volume Jobs = 6
+ Label Format = Inc-
+ Maximum Volumes = 7
+}
+
+Pool {
+ Name = Diff-Pool
+ Pool Type = Backup
+ Recycle = yes
+ AutoPrune = yes
+ Volume Retention = 40 days
+ Maximum Volume Jobs = 1
+ Label Format = Diff-
+ Maximum Volumes = 10
+}
+
+Messages {
+ Name = Standard
+ mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
+ -s \"Bacula: %t %e of %c %l\" %r"
+ operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
+ -s \"Bacula: Intervention needed for %j\" %r"
+ mail = root@domain.com = all, !skipped
+ operator = root@domain.com = mount
+ console = all, !skipped, !saved
+ append = "/home/bacula/bin/log" = all, !skipped
+}
+\end{verbatim}
+\normalsize
+
+and the Storage daemon's configuration file is:
+
+\footnotesize
+\begin{verbatim}
+Storage { # definition of myself
+ Name = bacula-sd
+ SDPort = 9103 # Director's port
+ WorkingDirectory = "/home/bacula/working"
+ Pid Directory = "/home/bacula/working"
+}
+Director {
+ Name = bacula-dir
+ Password = " *** CHANGE ME ***"
+}
+Device {
+ Name = FileStorage
+ Media Type = File
+ Archive Device = /files/bacula
+ LabelMedia = yes; # lets Bacula label unlabeled media
+ Random Access = Yes;
+ AutomaticMount = yes; # when device opened, read it
+ RemovableMedia = no;
+ AlwaysOpen = no;
+}
+Messages {
+ Name = Standard
+ director = bacula-dir = all
+}
+\end{verbatim}
+\normalsize
projects / bacula / docs / blobdiff