4 \section*{Using Pools to Manage Volumes}
5 \label{_ChapterStart11}
6 \index[general]{Volumes!Using Pools to Manage }
7 \index[general]{Using Pools to Manage Volumes }
8 \addcontentsline{toc}{section}{Using Pools to Manage Volumes}
10 If you manage 5 or 10 machines and have a nice tape backup, you don't need
11 Pools, and you may wonder what they are good for. In this chapter, you will
12 see that Pools can help you optimize disk storage space. The same techniques
13 can be applied to a shop that has multiple tape drives, or that wants to mount
14 various different Volumes to meet their needs.
16 The rest of this chapter will give an example involving backup to disk
17 Volumes, but most of the information applies equally well for tape Volumes.
20 \subsection*{The Problem}
21 \index[general]{Problem }
22 \addcontentsline{toc}{subsection}{Problem}
24 A site that I administer (a charitable organization) had a tape DDS-3 tape
25 drive that was failing. The exact reason for the failure is still unknown.
26 Worse yet, their full backup size is about 15GB whereas the capacity of their
27 broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the
28 necessary cassettes was more expensive than their budget could handle.
31 \subsection*{The Solution}
32 \index[general]{Solution }
33 \addcontentsline{toc}{subsection}{Solution}
35 They want to maintain 6 months of backup data, and be able to access the old
36 files on a daily basis for a week, a weekly basis for a month, then monthly
37 for 6 months. In addition, and offsite capability was not needed (well perhaps
38 it really is, but it was never used). Their daily changes amount to about
39 300MB on the average, or about 2GB per week.
41 As a consequence, the total volume of data they need to keep to meet their
42 needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
44 The chosen solution was to buy a 120GB hard disk for next to nothing -- far
45 less than 1/10th the price of a tape drive and the cassettes to handle the
46 same amount of data, and to have Bacula write to disk files.
48 The rest of this chapter will explain how to setup Bacula so that it would
49 automatically manage a set of disk files with the minimum intervention on my
50 part. The system has been running since 22 January 2004 until today (08 April
51 2004) with no intervention. Since we have not yet crossed the six month
52 boundary, we still lack some data to be sure the system performs as desired.
55 \subsection*{Overall Design}
56 \index[general]{Overall Design }
57 \index[general]{Design!Overall }
58 \addcontentsline{toc}{subsection}{Overall Design}
60 Getting Bacula to write to disk rather than tape in the simplest case is
61 rather easy, and is documented in the previous chapter. In addition, all the
62 directives discussed here are explained in that chapter. We'll leave it to you
63 to look at the details there. If you haven't read it and are not familiar with
64 Pools, you probably should at least read it once quickly for the ideas before
67 One needs to consider about what happens if we have only a single large Bacula
68 Volume defined on our hard disk. Everything works fine until the Volume fills,
69 then Bacula will ask you to mount a new Volume. This same problem applies to
70 the use of tape Volumes if your tape fills. Being a hard disk and the only one
71 you have, this will be a bit of a problem. It should be obvious that it is
72 better to use a number of smaller Volumes and arrange for Bacula to
73 automatically recycle them so that the disk storage space can be reused. The
74 other problem with a single Volume, is that at the current time (1.34.0)
75 Bacula does not seek within a disk Volume, so restoring a single file can take
76 more time than one would expect.
78 As mentioned, the solution is to have multiple Volumes, or files on the disk.
79 To do so, we need to limit the use and thus the size of a single Volume, by
80 time, by number of jobs, or by size. Any of these would work, but we chose to
81 limit the use of a single Volume by putting a single job in each Volume with
82 the exception of Volumes containing Incremental backup where there will be 6
83 jobs (a week's worth of data) per volume. The details of this will be
86 The next problem to resolve is recycling of Volumes. As you noted from above,
87 the requirements are to be able to restore monthly for 6 months, weekly for a
88 month, and daily for a week. So to simplify things, why not do a Full save
89 once a month, a Differential save once a week, and Incremental saves daily.
90 Now since each of these different kinds of saves needs to remain valid for
91 differing periods, the simplest way to do this (and possibly the only) is to
92 have a separate Pool for each backup type.
94 The decision was to use three Pools: one for Full saves, one for Differential
95 saves, and one for Incremental saves, and each would have a different number
96 of volumes and a different Retention period to accomplish the requirements.
99 \subsubsection*{Full Pool}
100 \index[general]{Pool!Full }
101 \index[general]{Full Pool }
102 \addcontentsline{toc}{subsubsection}{Full Pool}
104 Putting a single Full backup on each Volume, will require six Full save
105 Volumes, and a retention period of six months. The Pool needed to do that is:
114 Volume Retention = 6 months
115 Accept Any Volume = yes
116 Maximum Volume Jobs = 1
123 Since these are disk Volumes, no space is lost by having separate Volumes for
124 each backup (done once a month in this case). The items to note are the
125 retention period of six months (i.e. they are recycled after 6 months), that
126 there is one job per volume (Maximum Volume Jobs = 1), the volumes will be
127 labeled Full-0001, ... Full-0006 automatically. One could have labeled these
128 manual from the start, but why not use the features of Bacula.
131 \subsubsection*{Differential Pool}
132 \index[general]{Pool!Differential }
133 \index[general]{Differential Pool }
134 \addcontentsline{toc}{subsubsection}{Differential Pool}
136 For the Differential backup Pool, we choose a retention period of a bit longer
137 than a month and ensure that there is at least one Volume for each of the
138 maximum of five weeks in a month. So the following works:
147 Volume Retention = 40 days
148 Accept Any Volume = yes
149 Maximum Volume Jobs = 1
156 As you can see, the Differential Pool can grow to a maximum of six volumes,
157 and the Volumes are retained 40 days and there after can be recycled. Finally
158 there is one job per volume. This, of course, could be tightened up a lot, but
159 the expense here is a few GB which is not too serious.
162 \subsubsection*{Incremental Pool}
163 \index[general]{Incremental Pool }
164 \index[general]{Pool!Incremental }
165 \addcontentsline{toc}{subsubsection}{Incremental Pool}
167 Finally, here is the resource for the Incremental Pool:
176 Volume Retention = 20 days
177 Accept Any Volume = yes
178 Maximum Volume Jobs = 6
185 We keep the data for 20 days rather than just a week as the needs require. To
186 reduce the proliferation of volume names, we keep a week's worth of data (6
187 incremental backups) in each Volume. In practice, the retention period should
188 be set to just a bit more than a week and keep only two or three volumes
189 instead of five. Again, the lost is very little and as the system reaches the
190 full steady state, we can adjust these values so that the total disk usage
191 doesn't exceed the disk capacity.
194 \subsection*{The Actual Conf Files}
195 \index[general]{Files!Actual Conf }
196 \index[general]{Actual Conf Files }
197 \addcontentsline{toc}{subsection}{Actual Conf Files}
199 The following example shows you the actual files used, with only a few minor
200 modifications to simplify things.
202 The Director's configuration file is as follows:
206 Director { # define myself
209 QueryFile = "/home/bacula/bin/query.sql"
210 WorkingDirectory = "/home/bacula/working"
211 PidDirectory = "/home/bacula/working"
212 Maximum Concurrent Jobs = 1
216 # By default, this job will back up to disk in /tmp
222 Schedule = "WeeklyCycle"
226 Full Backup Pool = Full-Pool
227 Incremental Backup Pool = Inc-Pool
228 Differential Backup Pool = Diff-Pool
229 Write Bootstrap = "/home/bacula/working/client.bsr"
232 # List of files to be backed up
235 Include = signature=SHA1 compression=GZIP9 {
241 /proc /tmp /.journal /.fsck
246 Run = Full 1st sun at 1:05
247 Run = Differential 2nd-5th sun at 1:05
248 Run = Incremental mon-sat at 1:05
256 AutoPrune = yes # Prune expired Jobs/Files
257 Job Retention = 6 months
258 File Retention = 60 days
270 dbname = bacula; user = bacula; password = ""
275 Recycle = yes # automatically recycle Volumes
276 AutoPrune = yes # Prune expired volumes
277 Volume Retention = 6 months
278 Accept Any Volume = yes # write on any volume in the pool
279 Maximum Volume Jobs = 1
286 Recycle = yes # automatically recycle Volumes
287 AutoPrune = yes # Prune expired volumes
288 Volume Retention = 20 days
289 Accept Any Volume = yes
290 Maximum Volume Jobs = 6
299 Volume Retention = 40 days
300 Accept Any Volume = yes
301 Maximum Volume Jobs = 1
307 mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
308 -s \"Bacula: %t %e of %c %l\" %r"
309 operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
310 -s \"Bacula: Intervention needed for %j\" %r"
311 mail = root@domain.com = all, !skipped
312 operator = root@domain.com = mount
313 console = all, !skipped, !saved
314 append = "/home/bacula/bin/log" = all, !skipped
319 and the Storage daemon's configuration file is:
323 Storage { # definition of myself
325 SDPort = 9103 # Director's port
326 WorkingDirectory = "/home/bacula/working"
327 Pid Directory = "/home/bacula/working"
336 Archive Device = /files/bacula
337 LabelMedia = yes; # lets Bacula label unlabeled media
339 AutomaticMount = yes; # when device opened, read it
345 director = bacula-dir = all