4 \section*{Automated Disk Backup}
5 \label{_ChapterStart11}
6 \index[general]{Volumes!Using Pools to Manage}
7 \index[general]{Disk!Automated Backup}
8 \index[general]{Using Pools to Manage Volumes}
9 \index[general]{Automated Disk Backup}
10 \addcontentsline{toc}{section}{Using Pools to Manage Volumes}
11 \addcontentsline{toc}{section}{Automated Disk Backup}
13 If you manage 5 or 10 machines and have a nice tape backup, you don't need
14 Pools, and you may wonder what they are good for. In this chapter, you will
15 see that Pools can help you optimize disk storage space. The same techniques
16 can be applied to a shop that has multiple tape drives, or that wants to mount
17 various different Volumes to meet their needs.
19 The rest of this chapter will give an example involving backup to disk
20 Volumes, but most of the information applies equally well to tape Volumes.
23 \subsection*{The Problem}
24 \index[general]{Problem}
25 \addcontentsline{toc}{subsection}{Problem}
27 A site that I administer (a charitable organization) had a tape DDS-3 tape
28 drive that was failing. The exact reason for the failure is still unknown.
29 Worse yet, their full backup size is about 15GB whereas the capacity of their
30 broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the
31 necessary cassettes was more expensive than their budget could handle.
34 \subsection*{The Solution}
35 \index[general]{Solution}
36 \addcontentsline{toc}{subsection}{Solution}
38 They want to maintain 6 months of backup data, and be able to access the old
39 files on a daily basis for a week, a weekly basis for a month, then monthly
40 for 6 months. In addition, offsite capability was not needed (well perhaps
41 it really is, but it was never used). Their daily changes amount to about
42 300MB on the average, or about 2GB per week.
44 As a consequence, the total volume of data they need to keep to meet their
45 needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
47 The chosen solution was to buy a 120GB hard disk for next to nothing -- far
48 less than 1/10th the price of a tape drive and the cassettes to handle the
49 same amount of data, and to have Bacula write to disk files.
51 The rest of this chapter will explain how to setup Bacula so that it would
52 automatically manage a set of disk files with the minimum intervention on my
53 part. The system has been running since 22 January 2004 until today (17
54 September 2006) with no intervention, with the exception that I had to
55 add a second 120GB hard disk after a year because their needs grew
56 over that time to more than the 120GB (168GB to be exact). The only other
57 intervention I have made is a periodic (about once a year) Bacula upgrade.
60 \subsection*{Overall Design}
61 \index[general]{Overall Design}
62 \index[general]{Design!Overall}
63 \addcontentsline{toc}{subsection}{Overall Design}
65 Getting Bacula to write to disk rather than tape in the simplest case is
66 rather easy, and is documented in the previous chapter. In addition, all the
67 directives discussed here are explained in that chapter. We'll leave it to you
68 to look at the details there. If you haven't read it and are not familiar with
69 Pools, you probably should at least read it once quickly for the ideas before
72 One needs to consider about what happens if we have only a single large Bacula
73 Volume defined on our hard disk. Everything works fine until the Volume fills,
74 then Bacula will ask you to mount a new Volume. This same problem applies to
75 the use of tape Volumes if your tape fills. Being a hard disk and the only one
76 you have, this will be a bit of a problem. It should be obvious that it is
77 better to use a number of smaller Volumes and arrange for Bacula to
78 automatically recycle them so that the disk storage space can be reused. The
79 other problem with a single Volume, is that at the current time (1.34.0)
80 Bacula does not seek within a disk Volume, so restoring a single file can take
81 more time than one would expect.
83 As mentioned, the solution is to have multiple Volumes, or files on the disk.
84 To do so, we need to limit the use and thus the size of a single Volume, by
85 time, by number of jobs, or by size. Any of these would work, but we chose to
86 limit the use of a single Volume by putting a single job in each Volume with
87 the exception of Volumes containing Incremental backup where there will be 6
88 jobs (a week's worth of data) per volume. The details of this will be
91 The next problem to resolve is recycling of Volumes. As you noted from above,
92 the requirements are to be able to restore monthly for 6 months, weekly for a
93 month, and daily for a week. So to simplify things, why not do a Full save
94 once a month, a Differential save once a week, and Incremental saves daily.
95 Now since each of these different kinds of saves needs to remain valid for
96 differing periods, the simplest way to do this (and possibly the only) is to
97 have a separate Pool for each backup type.
99 The decision was to use three Pools: one for Full saves, one for Differential
100 saves, and one for Incremental saves, and each would have a different number
101 of volumes and a different Retention period to accomplish the requirements.
104 \subsubsection*{Full Pool}
105 \index[general]{Pool!Full}
106 \index[general]{Full Pool}
107 \addcontentsline{toc}{subsubsection}{Full Pool}
109 Putting a single Full backup on each Volume, will require six Full save
110 Volumes, and a retention period of six months. The Pool needed to do that is:
119 Volume Retention = 6 months
120 Maximum Volume Jobs = 1
127 Since these are disk Volumes, no space is lost by having separate Volumes for
128 each backup (done once a month in this case). The items to note are the
129 retention period of six months (i.e. they are recycled after 6 months), that
130 there is one job per volume (Maximum Volume Jobs = 1), the volumes will be
131 labeled Full-0001, ... Full-0006 automatically. One could have labeled these
132 manual from the start, but why not use the features of Bacula.
135 \subsubsection*{Differential Pool}
136 \index[general]{Pool!Differential}
137 \index[general]{Differential Pool}
138 \addcontentsline{toc}{subsubsection}{Differential Pool}
140 For the Differential backup Pool, we choose a retention period of a bit longer
141 than a month and ensure that there is at least one Volume for each of the
142 maximum of five weeks in a month. So the following works:
151 Volume Retention = 40 days
152 Maximum Volume Jobs = 1
159 As you can see, the Differential Pool can grow to a maximum of six volumes,
160 and the Volumes are retained 40 days and thereafter they can be recycled. Finally
161 there is one job per volume. This, of course, could be tightened up a lot, but
162 the expense here is a few GB which is not too serious.
165 \subsubsection*{Incremental Pool}
166 \index[general]{Incremental Pool}
167 \index[general]{Pool!Incremental}
168 \addcontentsline{toc}{subsubsection}{Incremental Pool}
170 Finally, here is the resource for the Incremental Pool:
179 Volume Retention = 20 days
180 Maximum Volume Jobs = 6
187 We keep the data for 20 days rather than just a week as the needs require. To
188 reduce the proliferation of volume names, we keep a week's worth of data (6
189 incremental backups) in each Volume. In practice, the retention period should
190 be set to just a bit more than a week and keep only two or three volumes
191 instead of five. Again, the lost is very little and as the system reaches the
192 full steady state, we can adjust these values so that the total disk usage
193 doesn't exceed the disk capacity.
196 \subsection*{The Actual Conf Files}
197 \index[general]{Files!Actual Conf}
198 \index[general]{Actual Conf Files}
199 \addcontentsline{toc}{subsection}{Actual Conf Files}
201 The following example shows you the actual files used, with only a few minor
202 modifications to simplify things.
204 The Director's configuration file is as follows:
208 Director { # define myself
211 QueryFile = "/home/bacula/bin/query.sql"
212 WorkingDirectory = "/home/bacula/working"
213 PidDirectory = "/home/bacula/working"
214 Maximum Concurrent Jobs = 1
215 Password = " *** CHANGE ME ***"
218 # By default, this job will back up to disk in /tmp
224 Schedule = "WeeklyCycle"
228 Full Backup Pool = Full-Pool
229 Incremental Backup Pool = Inc-Pool
230 Differential Backup Pool = Diff-Pool
231 Write Bootstrap = "/home/bacula/working/client.bsr"
235 # Backup the catalog database (after the nightly save)
237 Name = "BackupCatalog"
241 Schedule = "WeeklyCycleAfterBackup"
245 # This creates an ASCII copy of the catalog
246 RunBeforeJob = "/home/bacula/bin/make_catalog_backup bacula bacula"
247 # This deletes the copy of the catalog
248 RunAfterJob = "/home/bacula/bin/delete_catalog_backup"
249 Write Bootstrap = "/home/bacula/working/BackupCatalog.bsr"
250 Priority = 11 # run after main backup
253 # Standard Restore template, to be changed by Console program
255 Name = "RestoreFiles"
262 Where = /tmp/bacula-restores
267 # List of files to be backed up
270 Include = { Options { signature=SHA1; compression=GZIP9 }
288 Run = Full 1st sun at 2:05
289 Run = Differential 2nd-5th sun at 2:05
290 Run = Incremental mon-sat at 2:05
293 # This schedule does the catalog. It starts after the WeeklyCycle
295 Name = "WeeklyCycleAfterBackup"
296 Run = Full sun-sat at 2:10
299 # This is the backup of the catalog
302 Include { Options { signature=MD5 }
303 File = /home/bacula/working/bacula.sql
312 Password = " *** CHANGE ME ***"
313 AutoPrune = yes # Prune expired Jobs/Files
314 Job Retention = 6 months
315 File Retention = 60 days
322 Password = " *** CHANGE ME ***"
329 dbname = bacula; user = bacula; password = ""
335 Recycle = yes # automatically recycle Volumes
336 AutoPrune = yes # Prune expired volumes
337 Volume Retention = 6 months
338 Maximum Volume Jobs = 1
346 Recycle = yes # automatically recycle Volumes
347 AutoPrune = yes # Prune expired volumes
348 Volume Retention = 14 days
349 Maximum Volume Jobs = 6
359 Volume Retention = 35 days
360 Maximum Volume Jobs = 1
367 mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
368 -s \"Bacula: %t %e of %c %l\" %r"
369 operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
370 -s \"Bacula: Intervention needed for %j\" %r"
371 mail = root@domain.com = all, !skipped
372 operator = root@domain.com = mount
373 console = all, !skipped, !saved
374 append = "/home/bacula/bin/log" = all, !skipped
379 and the Storage daemon's configuration file is:
383 Storage { # definition of myself
385 SDPort = 9103 # Director's port
386 WorkingDirectory = "/home/bacula/working"
387 Pid Directory = "/home/bacula/working"
391 Password = " *** CHANGE ME ***"
396 Archive Device = /files/bacula
397 LabelMedia = yes; # lets Bacula label unlabeled media
399 AutomaticMount = yes; # when device opened, read it
405 director = bacula-dir = all