4 \chapter{Catalog Maintenance}
5 \label{CatMaintenanceChapter}
6 \index[general]{Maintenance!Catalog }
7 \index[general]{Catalog Maintenance }
9 Without proper setup and maintenance, your Catalog may continue to grow
10 indefinitely as you run Jobs and backup Files, and/or it may become
11 very inefficient and slow. How fast the size of your
12 Catalog grows depends on the number of Jobs you run and how many files they
13 backup. By deleting records within the database, you can make space available
14 for the new records that will be added during the next Job. By constantly
15 deleting old expired records (dates older than the Retention period), your
16 database size will remain constant.
18 If you started with the default configuration files, they already contain
19 reasonable defaults for a small number of machines (less than 5), so if you
20 fall into that case, catalog maintenance will not be urgent if you have a few
21 hundred megabytes of disk space free. Whatever the case may be, some knowledge
22 of retention periods will be useful.
25 \section{Setting Retention Periods}
26 \index[general]{Setting Retention Periods }
27 \index[general]{Periods!Setting Retention }
29 {\bf Bacula} uses three Retention periods: the {\bf File Retention} period,
30 the {\bf Job Retention} period, and the {\bf Volume Retention} period. Of
31 these three, the File Retention period is by far the most important in
32 determining how large your database will become.
34 The {\bf File Retention} and the {\bf Job Retention} are specified in each
35 Client resource as is shown below. The {\bf Volume Retention} period is
36 specified in the Pool resource, and the details are given in the next chapter
41 \item [File Retention = \lt{}time-period-specification\gt{}]
42 \index[dir]{File Retention }
43 The File Retention record defines the length of time that Bacula will keep
44 File records in the Catalog database. When this time period expires, and if
45 {\bf AutoPrune} is set to {\bf yes}, Bacula will prune (remove) File records
46 that are older than the specified File Retention period. The pruning will
47 occur at the end of a backup Job for the given Client. Note that the Client
48 database record contains a copy of the File and Job retention periods, but
49 Bacula uses the current values found in the Director's Client resource to do
52 Since File records in the database account for probably 80 percent of the
53 size of the database, you should carefully determine exactly what File
54 Retention period you need. Once the File records have been removed from
55 the database, you will no longer be able to restore individual files
56 in a Job. However, with Bacula version 1.37 and later, as long as the
57 Job record still exists, you will be able to restore all files in the
60 Retention periods are specified in seconds, but as a convenience, there are
61 a number of modifiers that permit easy specification in terms of minutes,
62 hours, days, weeks, months, quarters, or years on the record. See the
63 \ilink{ Configuration chapter}{Time} of this manual for additional details
64 of modifier specification.
66 The default File retention period is 60 days.
68 \item [Job Retention = \lt{}time-period-specification\gt{}]
69 \index[dir]{Job Retention }
70 The Job Retention record defines the length of time that {\bf Bacula}
71 will keep Job records in the Catalog database. When this time period
72 expires, and if {\bf AutoPrune} is set to {\bf yes} Bacula will prune
73 (remove) Job records that are older than the specified Job Retention
74 period. Note, if a Job record is selected for pruning, all associated File
75 and JobMedia records will also be pruned regardless of the File Retention
76 period set. As a consequence, you normally will set the File retention
77 period to be less than the Job retention period.
79 As mentioned above, once the File records are removed from the database,
80 you will no longer be able to restore individual files from the Job.
81 However, as long as the Job record remains in the database, you will be
82 able to restore all the files backuped for the Job (on version 1.37 and
83 later). As a consequence, it is generally a good idea to retain the Job
84 records much longer than the File records.
86 The retention period is specified in seconds, but as a convenience, there
87 are a number of modifiers that permit easy specification in terms of
88 minutes, hours, days, weeks, months, quarters, or years. See the \ilink{
89 Configuration chapter}{Time} of this manual for additional details of
90 modifier specification.
92 The default Job Retention period is 180 days.
94 \item [AutoPrune = \lt{}yes/no\gt{}]
95 \index[dir]{AutoPrune }
96 If AutoPrune is set to {\bf yes} (default), Bacula will automatically apply
97 the File retention period and the Job retention period for the Client at the
100 If you turn this off by setting it to {\bf no}, your Catalog will grow each
104 \label{CompactingMySQL}
105 \section{Compacting Your MySQL Database}
106 \index[general]{Database!Compacting Your MySQL }
107 \index[general]{Compacting Your MySQL Database }
109 Over time, as noted above, your database will tend to grow. I've noticed that
110 even though Bacula regularly prunes files, {\bf MySQL} does not effectively
111 use the space, and instead continues growing. To avoid this, from time to
112 time, you must compact your database. Normally, large commercial database such
113 as Oracle have commands that will compact a database to reclaim wasted file
114 space. MySQL has the {\bf OPTIMIZE TABLE} command that you can use, and SQLite
115 version 2.8.4 and greater has the {\bf VACUUM} command. We leave it to you to
116 explore the utility of the {\bf OPTIMIZE TABLE} command in MySQL.
118 All database programs have some means of writing the database out in ASCII
119 format and then reloading it. Doing so will re-create the database from
120 scratch producing a compacted result, so below, we show you how you can do
121 this for MySQL, PostgreSQL and SQLite.
123 For a {\bf MySQL} database, you could write the Bacula database as an ASCII
124 file (bacula.sql) then reload it by doing the following:
128 mysqldump -f --opt bacula > bacula.sql
129 mysql bacula < bacula.sql
134 Depending on the size of your database, this will take more or less time and a
135 fair amount of disk space. For example, if I cd to the location of the MySQL
136 Bacula database (typically /opt/mysql/var or something similar) and enter:
144 I get {\bf 620,644} which means there are that many blocks containing 1024
145 bytes each or approximately 635 MB of data. After doing the {\bf mysqldump}, I
146 had a bacula.sql file that had {\bf 174,356} blocks, and after doing the {\bf
147 mysql} command to recreate the database, I ended up with a total of {\bf
148 210,464} blocks rather than the original {\bf 629,644}. In other words, the
149 compressed version of the database took approximately one third of the space
150 of the database that had been in use for about a year.
152 As a consequence, I suggest you monitor the size of your database and from
153 time to time (once every six months or year), compress it.
155 \label{DatabaseRepair}
156 \label{RepairingMySQL}
157 \section{Repairing Your MySQL Database}
158 \index[general]{Database!Repairing Your MySQL }
159 \index[general]{Repairing Your MySQL Database }
161 If you find that you are getting errors writing to your MySQL database, or
162 Bacula hangs each time it tries to access the database, you should consider
163 running MySQL's database check and repair routines. The program you need to
164 run depends on the type of database indexing you are using. If you are using
165 the default, you will probably want to use {\bf myisamchk}. For more details
166 on how to do this, please consult the MySQL document at:
168 http://www.mysql.com/doc/en/Repair.html}
169 {http://www.mysql.com/doc/en/Repair.html}.
171 If the errors you are getting are simply SQL warnings, then you might try
172 running dbcheck before (or possibly after) using the MySQL database repair
173 program. It can clean up many of the orphaned record problems, and certain
174 other inconsistencies in the Bacula database.
176 A typical cause of MySQL database problems is if your partition fills. In
177 such a case, you will need to create additional space on the partition or
178 free up some space then repair the database probably using {\bf myisamchk}.
179 Recently my root partition filled and the MySQL database was corrupted.
180 Simply running {\bf myisamchk -r} did not fix the problem. However,
181 the following script did the trick for me:
188 t=`echo $i | cut -f 1 -d '.' -`
189 mysql bacula <<END_OF_DATA
195 chown mysql:mysql ${i}
201 I invoked it with the following commands:
205 cd /var/lib/mysql/bacula
210 Then after ensuring that the database was correctly fixed, I did:
213 cd /var/lib/mysql/bacula
218 \section{MySQL Table is Full}
219 \index[general]{Database!MySQL Table is Full}
220 \index[general]{MySQL Table is Full}
222 If you are running into the error {\bf The table 'File' is full ...},
223 it is probably because on version 4.x MySQL, the table is limited by
224 default to a maximum size of 4 GB and you have probably run into
225 the limit. The solution can be found at:
226 \elink{http://dev.mysql.com/doc/refman/5.0/en/full-table.html}
227 {http://dev.mysql.com/doc/refman/5.0/en/full-table.html}
229 You can display the maximum length of your table with:
234 SHOW TABLE STATUS FROM bacula like "File";
238 If the column labeled "Max\_data\_length" is around 4Gb, this is likely
239 to be the source of your problem, and you can modify it with:
244 ALTER TABLE File MAX_ROWS=281474976710656;
248 Alternatively you can modify your /etc/my.conf file before creating the
249 Bacula tables, and in the [mysqld] section set:
253 set-variable = myisam_data_pointer_size=6
257 The above myisam data pointer size must be made before you create your
258 Bacula tables or it will have no effect.
260 The row and pointer size changes should already be the default on MySQL
261 version 5.x, so making these changes should only be necessary on MySQL 4.x
262 depending on the size of your catalog database.
264 \section{MySQL Server Has Gone Away}
265 \index[general]{Database!MySQL Server Has Gone Away}
266 \index[general]{MySQL Server Has Gone Away}
267 If you are having problems with the MySQL server disconnecting or with
268 messages saying that your MySQL server has gone away, then please read
269 the MySQL documentation, which can be found at:
271 \elink{http://dev.mysql.com/doc/refman/5.0/en/gone-away.html}
272 {http://dev.mysql.com/doc/refman/5.0/en/gone-away.html}
275 \label{RepairingPSQL}
276 \section{Repairing Your PostgreSQL Database}
277 \index[general]{Database!Repairing Your PostgreSQL }
278 \index[general]{Repairing Your PostgreSQL Database }
280 The same considerations apply that are indicated above for MySQL. That is,
281 consult the PostgreSQL documents for how to repair the database, and also
282 consider using Bacula's dbcheck program if the conditions are reasonable for
285 \label{DatabasePerformance}
286 \section{Database Performance Issues}
287 \index[general]{Database Performance Issues}
288 \index[general]{Performance!Database}
290 There are a considerable number of ways each of the databases can be
291 tuned to improve the performance. Going from an untuned database to one
292 that is properly tuned can make a difference of a factor of 100 or more
293 in the time to insert or search for records.
295 For each of the databases, you may get significant improvements by adding
296 additional indexes. The comments in the Bacula make\_xxx\_tables give some
297 indications as to what indexes may be appropriate. Please see below
298 for specific instructions on checking indexes.
300 For MySQL, what is very important is to use the examine the
301 my.cnf file (usually in /etc/my.cnf).
302 You may obtain significant performances by switching to
303 the my-large.cnf or my-huge.cnf files that come with the MySQL source
306 For SQLite3, one significant factor in improving the performance is
307 to ensure that there is a "PRAGMA synchronous = NORMAL;" statement.
308 This reduces the number of times that the database flushes the in memory
309 cache to disk. There are other settings for this PRAGMA that can
310 give even further performance improvements at the risk of a database
311 corruption if your system crashes.
313 For PostgreSQL, you might want to consider turning fsync off. Of course
314 doing so can cause corrupted databases in the event of a machine crash.
315 There are many different ways that you can tune PostgreSQL, the
316 following document discusses a few of them:
318 http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html}
319 {http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html}.
321 There is also a PostgreSQL FAQ question number 3.3 that may
322 answer some of your questions about how to improve performance
323 of the PostgreSQL engine:
325 http://www.postgresql.org/docs/faqs.FAQ.html\#3.3}
326 {http://www.postgresql.org/docs/faqs.FAQ.html\#3.3}.
327 % TODO: verify above is correct. is this okay for book?
329 Also for PostgreSQL, look at what "effective\_cache\_size". For a 2GB memory
330 machine, you probably want to set it at 131072, but don't set it too high.
331 In addition, for a 2GB system, work\_mem = 256000 and
332 maintenance\_work\_mem = 256000 seem to be reasonable values. Make
333 sure your checkpoint\_segments is set to at least 8.
337 \section{Performance Issues Indexes}
338 \index[general]{Database Performance Issues Indexes}
339 \index[general]{Performance!Database}
340 One of the most important considerations for improving performance on
341 the Bacula database is to ensure that it has all the appropriate indexes.
342 Several users have reported finding that their database did not have
343 all the indexes in the default configuration. In addition, you may
344 find that because of your own usage patterns, you need additional indexes.
346 The most important indexes for performance are the three indexes on the
347 {\bf File} table. The first index is on {\bf FileId} and is automatically
348 made because it is the unique key used to access the table. The other
349 two are the JobId index and the (Filename, PathId) index. If these Indexes
350 are not present, your performance may suffer a lot.
352 \subsection{PostgreSQL Indexes}
353 On PostgreSQL, you can check to see if you have the proper indexes using
354 the following commands:
359 select * from pg_indexes where tablename='file';
363 If you do not see output that indicates that all three indexes
364 are created, you can create the two additional indexes using:
369 CREATE INDEX file_jobid_idx on file (jobid);
370 CREATE INDEX file_fp_idx on file (filenameid, pathid);
374 \subsection{MySQL Indexes}
375 On MySQL, you can check if you have the proper indexes by:
380 show index from File;
384 If the indexes are not present, especially the JobId index, you can
385 create them with the following commands:
390 CREATE INDEX file_jobid_idx on File (JobId);
391 CREATE INDEX file_jpf_idx on File (Job, FilenameId, PathId);
395 Though normally not a problem, you should ensure that the indexes
396 defined for Filename and Path are both set to 255 characters. Some users
397 reported performance problems when their indexes were set to 50 characters.
403 show index from Filename;
404 show index from Path;
408 and what is important is that for Filename, you have an index with
409 Key\_name "Name" and Sub\_part "255". For Pth, you should have a Key\_name
410 "Path" and Sub\_part "255". If one or the other does not exist or the
411 Sub\_part is less that 255, you can drop and recreate the appropriate
417 DROP INDEX Path on Path;
418 CREATE INDEX Path on Path (Path(255);
420 DROP INDEX Name on Filename;
421 CREATE INDEX Name on Filename (Name(255));
426 \subsection{SQLite Indexes}
427 On SQLite, you can check if you have the proper indexes by:
431 sqlite <path>bacula.db
432 select * from sqlite_master where type='index' and tbl_name='File';
436 If the indexes are not present, especially the JobId index, you can
437 create them with the following commands:
442 CREATE INDEX file_jobid_idx on File (JobId);
443 CREATE INDEX file_jfp_idx on File (Job, FilenameId, PathId);
449 \label{CompactingPostgres}
450 \section{Compacting Your PostgreSQL Database}
451 \index[general]{Database!Compacting Your PostgreSQL }
452 \index[general]{Compacting Your PostgreSQL Database }
454 Over time, as noted above, your database will tend to grow. I've noticed that
455 even though Bacula regularly prunes files, PostgreSQL has a {\bf VACUUM}
456 command that will compact your database for you. Alternatively you may want to
457 use the {\bf vacuumdb} command, which can be run from a cron job.
459 All database programs have some means of writing the database out in ASCII
460 format and then reloading it. Doing so will re-create the database from
461 scratch producing a compacted result, so below, we show you how you can do
464 For a {\bf PostgreSQL} database, you could write the Bacula database as an
465 ASCII file (bacula.sql) then reload it by doing the following:
469 pg_dump -c bacula > bacula.sql
470 cat bacula.sql | psql bacula
475 Depending on the size of your database, this will take more or less time and a
476 fair amount of disk space. For example, you can {\bf cd} to the location of
477 the Bacula database (typically /usr/local/pgsql/data or possible
478 /var/lib/pgsql/data) and check the size.
480 There are certain PostgreSQL users who do not recommend the above
481 procedure. They have the following to say:
483 need to be dumped/restored to keep the database efficient. A normal
484 process of vacuuming will prevent the database from every getting too
485 large. If you want to fine-tweak the database storage, commands such
486 as VACUUM FULL, REINDEX, and CLUSTER exist specifically to keep you
487 from having to do a dump/restore.
489 Finally, you might want to look at the PostgreSQL documentation on
491 \elink{http://www.postgresql.org/docs/8.1/interactive/maintenance.html}
492 {http://www.postgresql.org/docs/8.1/interactive/maintenance.html}.
494 \section{Compacting Your SQLite Database}
495 \index[general]{Compacting Your SQLite Database }
496 \index[general]{Database!Compacting Your SQLite }
498 First please read the previous section that explains why it is necessary to
499 compress a database. SQLite version 2.8.4 and greater have the {\bf Vacuum}
500 command for compacting the database.
504 cd {\bf working-directory}
505 echo 'vacuum;' | sqlite bacula.db
509 As an alternative, you can use the following commands, adapted to your system:
514 cd {\bf working-directory}
515 echo '.dump' | sqlite bacula.db > bacula.sql
517 sqlite bacula.db < bacula.sql
522 Where {\bf working-directory} is the directory that you specified in the
523 Director's configuration file. Note, in the case of SQLite, it is necessary to
524 completely delete (rm) the old database before creating a new compressed
527 \section{Migrating from SQLite to MySQL}
528 \index[general]{MySQL!Migrating from SQLite to }
529 \index[general]{Migrating from SQLite to MySQL }
531 You may begin using Bacula with SQLite then later find that you want to switch
532 to MySQL for any of a number of reasons: SQLite tends to use more disk than
533 MySQL; when the database is corrupted it is often more catastrophic than
534 with MySQL or PostgreSQL.
535 Several users have succeeded in converting from SQLite to MySQL by
536 exporting the MySQL data and then processing it with Perl scripts
537 prior to putting it into MySQL. This is, however, not a simple
540 \label{BackingUpBacula}
541 \section{Backing Up Your Bacula Database}
542 \index[general]{Backing Up Your Bacula Database }
543 \index[general]{Database!Backing Up Your Bacula }
545 If ever the machine on which your Bacula database crashes, and you need to
546 restore from backup tapes, one of your first priorities will probably be to
547 recover the database. Although Bacula will happily backup your catalog
548 database if it is specified in the FileSet, this is not a very good way to do
549 it, because the database will be saved while Bacula is modifying it. Thus the
550 database may be in an instable state. Worse yet, you will backup the database
551 before all the Bacula updates have been applied.
553 To resolve these problems, you need to backup the database after all the backup
554 jobs have been run. In addition, you will want to make a copy while Bacula is
555 not modifying it. To do so, you can use two scripts provided in the release
556 {\bf make\_catalog\_backup} and {\bf delete\_catalog\_backup}. These files
557 will be automatically generated along with all the other Bacula scripts. The
558 first script will make an ASCII copy of your Bacula database into {\bf
559 bacula.sql} in the working directory you specified in your configuration, and
560 the second will delete the {\bf bacula.sql} file.
562 The basic sequence of events to make this work correctly is as follows:
565 \item Run all your nightly backups
566 \item After running your nightly backups, run a Catalog backup Job
567 \item The Catalog backup job must be scheduled after your last nightly backup
569 \item You use {\bf RunBeforeJob} to create the ASCII backup file and {\bf
570 RunAfterJob} to clean up
573 Assuming that you start all your nightly backup jobs at 1:05 am (and that they
574 run one after another), you can do the catalog backup with the following
575 additional Director configuration statements:
579 # Backup the catalog database (after the nightly save)
581 Name = "BackupCatalog"
585 Schedule = "WeeklyCycleAfterBackup"
589 RunBeforeJob = "/home/kern/bacula/bin/make_catalog_backup"
590 RunAfterJob = "/home/kern/bacula/bin/delete_catalog_backup"
591 Write Bootstrap = "/home/kern/bacula/working/BackupCatalog.bsr"
593 # This schedule does the catalog. It starts after the WeeklyCycle
595 Name = "WeeklyCycleAfterBackup
596 Run = Level=Full sun-sat at 1:10
598 # This is the backup of the catalog
605 File = \lt{}working_directory\gt{}/bacula.sql
611 Be sure to write a bootstrap file as in the above example. However, it is preferable
612 to write or copy the bootstrap file to another computer. It will allow
613 you to quickly recover the database backup should that be necessary. If
614 you do not have a bootstrap file, it is still possible to recover your
615 database backup, but it will be more work and take longer.
617 \label{BackingUPOtherDBs}
618 \section{Backing Up Third Party Databases}
619 \index[general]{Backing Up Third Party Databases }
620 \index[general]{Databases!Backing Up Third Party }
622 If you are running a database in production mode on your machine, Bacula will
623 happily backup the files, but if the database is in use while Bacula is
624 reading it, you may back it up in an unstable state.
626 The best solution is to shutdown your database before backing it up, or use
627 some tool specific to your database to make a valid live copy perhaps by
628 dumping the database in ASCII format. I am not a database expert, so I cannot
629 provide you advice on how to do this, but if you are unsure about how to
630 backup your database, you might try visiting the Backup Central site, which
631 has been renamed Storage Mountain (www.backupcentral.com). In particular,
633 \elink{ Free Backup and Recovery
634 Software}{http://www.backupcentral.com/toc-free-backup-software.html} page has
635 links to scripts that show you how to shutdown and backup most major
639 \section{Database Size}
640 \index[general]{Size!Database }
641 \index[general]{Database Size }
643 As mentioned above, if you do not do automatic pruning, your Catalog will grow
644 each time you run a Job. Normally, you should decide how long you want File
645 records to be maintained in the Catalog and set the {\bf File Retention}
646 period to that time. Then you can either wait and see how big your Catalog
647 gets or make a calculation assuming approximately 154 bytes for each File
648 saved and knowing the number of Files that are saved during each backup and
649 the number of Clients you backup.
651 For example, suppose you do a backup of two systems, each with 100,000 files.
652 Suppose further that you do a Full backup weekly and an Incremental every day,
653 and that the Incremental backup typically saves 4,000 files. The size of your
654 database after a month can roughly be calculated as:
658 Size = 154 * No. Systems * (100,000 * 4 + 10,000 * 26)
662 where we have assumed four weeks in a month and 26 incremental backups per month.
663 This would give the following:
667 Size = 154 * 2 * (100,000 * 4 + 10,000 * 26)
669 Size = 308 * (400,000 + 260,000)
671 Size = 203,280,000 bytes
675 So for the above two systems, we should expect to have a database size of
676 approximately 200 Megabytes. Of course, this will vary according to how many
677 files are actually backed up.
679 Below are some statistics for a MySQL database containing Job records for five
680 Clients beginning September 2001 through May 2002 (8.5 months) and File
681 records for the last 80 days. (Older File records have been pruned). For these
682 systems, only the user files and system files that change are backed up. The
683 core part of the system is assumed to be easily reloaded from the Red Hat rpms.
686 In the list below, the files (corresponding to Bacula Tables) with the
687 extension .MYD contain the data records whereas files with the extension .MYI
690 You will note that the File records (containing the file attributes) make up
691 the large bulk of the number of records as well as the space used (459 Mega
692 Bytes including the indexes). As a consequence, the most important Retention
693 period will be the {\bf File Retention} period. A quick calculation shows that
694 for each File that is saved, the database grows by approximately 150 bytes.
700 ============ ========= ===========
703 344,394,684 3,080,191 File.MYD
705 2,590,316 106,902 Filename.MYD
706 3,026,944 Filename.MYI
709 49,062 1,326 JobMedia.MYD
711 141,752 1,378 Job.MYD
715 1,299,512 22,233 Path.MYD
724 This database has a total size of approximately 450 Megabytes.
726 If we were using SQLite, the determination of the total database size would be
727 much easier since it is a single file, but we would have less insight to the
728 size of the individual tables as we have in this case.
730 Note, SQLite databases may be as much as 50\% larger than MySQL databases due
731 to the fact that all data is stored as ASCII strings. That is even binary
732 integers are stored as ASCII strings, and this seems to increase the space