4 \chapter{Automated Disk Backup}
6 \index[general]{Volumes!Using Pools to Manage}
7 \index[general]{Disk!Automated Backup}
8 \index[general]{Using Pools to Manage Volumes}
9 \index[general]{Automated Disk Backup}
11 If you manage five or ten machines and have a nice tape backup, you don't need
12 Pools, and you may wonder what they are good for. In this chapter, you will
13 see that Pools can help you optimize disk storage space. The same techniques
14 can be applied to a shop that has multiple tape drives, or that wants to mount
15 various different Volumes to meet their needs.
17 The rest of this chapter will give an example involving backup to disk
18 Volumes, but most of the information applies equally well to tape Volumes.
22 \index[general]{Problem}
24 A site that I administer (a charitable organization) had a tape DDS-3 tape
25 drive that was failing. The exact reason for the failure is still unknown.
26 Worse yet, their full backup size is about 15GB whereas the capacity of their
27 broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the
28 necessary cassettes was more expensive than their budget could handle.
31 \section{The Solution}
32 \index[general]{Solution}
34 They want to maintain six months of backup data, and be able to access the old
35 files on a daily basis for a week, a weekly basis for a month, then monthly
36 for six months. In addition, offsite capability was not needed (well perhaps
37 it really is, but it was never used). Their daily changes amount to about
38 300MB on the average, or about 2GB per week.
40 As a consequence, the total volume of data they need to keep to meet their
41 needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
43 The chosen solution was to buy a 120GB hard disk for next to nothing -- far
44 less than 1/10th the price of a tape drive and the cassettes to handle the
45 same amount of data, and to have Bacula write to disk files.
47 The rest of this chapter will explain how to setup Bacula so that it would
48 automatically manage a set of disk files with the minimum sysadmin
49 intervention. The system has been running since 22 January 2004 until today
50 (23 June 2007) with no intervention, with the exception of adding
51 a second 120GB hard disk after a year because their needs grew
52 over that time to more than the 120GB (168GB to be exact). The only other
53 intervention I have made is a periodic (about once a year) Bacula upgrade.
56 \section{Overall Design}
57 \index[general]{Overall Design}
58 \index[general]{Design!Overall}
60 Getting Bacula to write to disk rather than tape in the simplest case is
61 rather easy, and is documented in the previous chapter. In addition, all the
62 directives discussed here are explained in that chapter. We'll leave it to you
63 to look at the details there. If you haven't read it and are not familiar with
64 Pools, you probably should at least read it once quickly for the ideas before
67 One needs to consider about what happens if we have only a single large Bacula
68 Volume defined on our hard disk. Everything works fine until the Volume fills,
69 then Bacula will ask you to mount a new Volume. This same problem applies to
70 the use of tape Volumes if your tape fills. Being a hard disk and the only one
71 you have, this will be a bit of a problem. It should be obvious that it is
72 better to use a number of smaller Volumes and arrange for Bacula to
73 automatically recycle them so that the disk storage space can be reused. The
74 other problem with a single Volume, is that until version 2.0.0,
75 Bacula did not seek within a disk Volume, so restoring a single file can take
76 more time than one would expect.
78 As mentioned, the solution is to have multiple Volumes, or files on the disk.
79 To do so, we need to limit the use and thus the size of a single Volume, by
80 time, by number of jobs, or by size. Any of these would work, but we chose to
81 limit the use of a single Volume by putting a single job in each Volume with
82 the exception of Volumes containing Incremental backup where there will be 6
83 jobs (a week's worth of data) per volume. The details of this will be
84 discussed shortly. This is a single client backup, so if you have multiple
85 clients you will need to multiply those numbers by the number of clients,
86 or use a different system for switching volumes, such as limiting the
89 The next problem to resolve is recycling of Volumes. As you noted from above,
90 the requirements are to be able to restore monthly for 6 months, weekly for a
91 month, and daily for a week. So to simplify things, why not do a Full save
92 once a month, a Differential save once a week, and Incremental saves daily.
93 Now since each of these different kinds of saves needs to remain valid for
94 differing periods, the simplest way to do this (and possibly the only) is to
95 have a separate Pool for each backup type.
97 The decision was to use three Pools: one for Full saves, one for Differential
98 saves, and one for Incremental saves, and each would have a different number
99 of volumes and a different Retention period to accomplish the requirements.
102 \subsection{Full Pool}
103 \index[general]{Pool!Full}
104 \index[general]{Full Pool}
106 Putting a single Full backup on each Volume, will require six Full save
107 Volumes, and a retention period of six months. The Pool needed to do that is:
116 Volume Retention = 6 months
117 Maximum Volume Jobs = 1
124 Since these are disk Volumes, no space is lost by having separate Volumes for
125 each backup (done once a month in this case). The items to note are the
126 retention period of six months (i.e. they are recycled after six months), that
127 there is one job per volume (Maximum Volume Jobs = 1), the volumes will be
128 labeled Full-0001, ... Full-0006 automatically. One could have labeled these
129 manually from the start, but why not use the features of Bacula.
131 Six months after the first volume is used, it will be subject to pruning
132 and thus recycling, so with a maximum of 9 volumes, there should always be
133 3 volumes available (note, they may all be marked used, but they will be
134 marked purged and recycled as needed).
136 If you have two clients, you would want to set {\bf Maximum Volume Jobs} to
137 2 instead of one, or set a limit on the size of the Volumes, and possibly
138 increase the maximum number of Volumes.
142 \subsection{Differential Pool}
143 \index[general]{Pool!Differential}
144 \index[general]{Differential Pool}
146 For the Differential backup Pool, we choose a retention period of a bit longer
147 than a month and ensure that there is at least one Volume for each of the
148 maximum of five weeks in a month. So the following works:
157 Volume Retention = 40 days
158 Maximum Volume Jobs = 1
165 As you can see, the Differential Pool can grow to a maximum of 9 volumes,
166 and the Volumes are retained 40 days and thereafter they can be recycled. Finally
167 there is one job per volume. This, of course, could be tightened up a lot, but
168 the expense here is a few GB which is not too serious.
170 If a new volume is used every week, after 40 days, one will have used 7
171 volumes, and there should then always be 3 volumes that can be purged and
174 See the discussion above concering the Full pool for how to handle multiple
178 \subsection{Incremental Pool}
179 \index[general]{Incremental Pool}
180 \index[general]{Pool!Incremental}
182 Finally, here is the resource for the Incremental Pool:
191 Volume Retention = 20 days
192 Maximum Volume Jobs = 6
199 We keep the data for 20 days rather than just a week as the needs require. To
200 reduce the proliferation of volume names, we keep a week's worth of data (6
201 incremental backups) in each Volume. In practice, the retention period should
202 be set to just a bit more than a week and keep only two or three volumes
203 instead of five. Again, the lost is very little and as the system reaches the
204 full steady state, we can adjust these values so that the total disk usage
205 doesn't exceed the disk capacity.
207 If you have two clients, the simplest thing to do is to increase the
208 maximum volume jobs from 6 to 12. As mentioned above, it is also possible
209 limit the size of the volumes. However, in that case, you will need to
210 have a better idea of the volume or add sufficient volumes to the pool so
211 that you will be assured that in the next cycle (after 20 days) there is
212 at least one volume that is pruned and can be recycled.
216 \section{The Actual Conf Files}
217 \index[general]{Files!Actual Conf}
218 \index[general]{Actual Conf Files}
220 The following example shows you the actual files used, with only a few minor
221 modifications to simplify things.
223 The Director's configuration file is as follows:
227 Director { # define myself
230 QueryFile = "/home/bacula/bin/query.sql"
231 WorkingDirectory = "/home/bacula/working"
232 PidDirectory = "/home/bacula/working"
233 Maximum Concurrent Jobs = 1
234 Password = " *** CHANGE ME ***"
237 # By default, this job will back up to disk in /tmp
243 Schedule = "WeeklyCycle"
247 Full Backup Pool = Full-Pool
248 Incremental Backup Pool = Inc-Pool
249 Differential Backup Pool = Diff-Pool
250 Write Bootstrap = "/home/bacula/working/client.bsr"
254 # Backup the catalog database (after the nightly save)
256 Name = "BackupCatalog"
260 Schedule = "WeeklyCycleAfterBackup"
264 # This creates an ASCII copy of the catalog
265 # WARNING!!! Passing the password via the command line is insecure.
266 # see comments in make_catalog_backup for details.
267 RunBeforeJob = "/home/bacula/bin/make_catalog_backup bacula bacula"
268 # This deletes the copy of the catalog
269 RunAfterJob = "/home/bacula/bin/delete_catalog_backup"
270 Write Bootstrap = "/home/bacula/working/BackupCatalog.bsr"
271 Priority = 11 # run after main backup
274 # Standard Restore template, to be changed by Console program
276 Name = "RestoreFiles"
283 Where = /tmp/bacula-restores
288 # List of files to be backed up
291 Include = { Options { signature=SHA1; compression=GZIP9 }
309 Run = Level=Full 1st sun at 2:05
310 Run = Level=Differential 2nd-5th sun at 2:05
311 Run = Level=Incremental mon-sat at 2:05
314 # This schedule does the catalog. It starts after the WeeklyCycle
316 Name = "WeeklyCycleAfterBackup"
317 Run = Level=Full sun-sat at 2:10
320 # This is the backup of the catalog
323 Include { Options { signature=MD5 }
324 File = /home/bacula/working/bacula.sql
333 Password = " *** CHANGE ME ***"
334 AutoPrune = yes # Prune expired Jobs/Files
335 Job Retention = 6 months
336 File Retention = 60 days
343 Password = " *** CHANGE ME ***"
350 dbname = bacula; user = bacula; password = ""
356 Recycle = yes # automatically recycle Volumes
357 AutoPrune = yes # Prune expired volumes
358 Volume Retention = 6 months
359 Maximum Volume Jobs = 1
367 Recycle = yes # automatically recycle Volumes
368 AutoPrune = yes # Prune expired volumes
369 Volume Retention = 20 days
370 Maximum Volume Jobs = 6
380 Volume Retention = 40 days
381 Maximum Volume Jobs = 1
388 mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
389 -s \"Bacula: %t %e of %c %l\" %r"
390 operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
391 -s \"Bacula: Intervention needed for %j\" %r"
392 mail = root@domain.com = all, !skipped
393 operator = root@domain.com = mount
394 console = all, !skipped, !saved
395 append = "/home/bacula/bin/log" = all, !skipped
400 and the Storage daemon's configuration file is:
404 Storage { # definition of myself
406 SDPort = 9103 # Director's port
407 WorkingDirectory = "/home/bacula/working"
408 Pid Directory = "/home/bacula/working"
412 Password = " *** CHANGE ME ***"
417 Archive Device = /files/bacula
418 LabelMedia = yes; # lets Bacula label unlabeled media
420 AutomaticMount = yes; # when device opened, read it
426 director = bacula-dir = all