From 0da8b68a3e9baea986b5284bb163c4cad7a35436 Mon Sep 17 00:00:00 2001 From: Kern Sibbald Date: Sun, 17 Sep 2006 20:32:15 +0000 Subject: [PATCH] Update --- docs/manual/catmaintenance.tex | 106 +++++++++++++++++++++++--- docs/manual/filedconf.tex | 6 +- docs/manual/pools.tex | 133 ++++++++++++++++++++++++--------- docs/manual/progs.tex | 1 - docs/manual/recycling.tex | 1 + docs/manual/restore.tex | 29 +++++-- docs/manual/version.tex | 2 +- 7 files changed, 225 insertions(+), 53 deletions(-) diff --git a/docs/manual/catmaintenance.tex b/docs/manual/catmaintenance.tex index f9bf6913..b23dbc59 100644 --- a/docs/manual/catmaintenance.tex +++ b/docs/manual/catmaintenance.tex @@ -8,7 +8,8 @@ \addcontentsline{toc}{section}{Catalog Maintenance} Without proper setup and maintenance, your Catalog may continue to grow -indefinitely as you run Jobs and backup Files. How fast the size of your +indefinitely as you run Jobs and backup Files, and/or it may become +very inefficient and slow. How fast the size of your Catalog grows depends on the number of Jobs you run and how many files they backup. By deleting records within the database, you can make space available for the new records that will be added during the next Job. By constantly @@ -154,8 +155,9 @@ of the database that had been in use for about a year. As a consequence, I suggest you monitor the size of your database and from time to time (once every 6 months or year), compress it. -\label{RepairingMySQL} +\label{DatabaseRepair} +\label{RepairingMySQL} \subsection*{Repairing Your MySQL Database} \index[general]{Database!Repairing Your MySQL } \index[general]{Repairing Your MySQL Database } @@ -175,8 +177,48 @@ If the errors you are getting are simply SQL warnings, then you might try running dbcheck before (or possibly after) using the MySQL database repair program. It can clean up many of the orphaned record problems, and certain other inconsistencies in the Bacula database. -\label{RepairingPSQL} +If you are running into the error {\bf The table 'File' is full ...}, +it is probably because on version 4.x MySQL, the table is limited by +default to a maximum size of 4 GB and you have probably run into +the limit. The solution can be found at: +\elink{http://dev.mysql.com/doc/refman/5.0/en/full-table.html} +{http://dev.mysql.com/doc/refman/5.0/en/full-table.html} + +You can display the maximum length of your table with: + +\footnotesize +\begin{verbatim} +mysql bacula +SHOW TABLE STATUS FROM bacula like "File"; +\end{verbatim} +\normalsize + +If the column labeld "Max_data_length" is around 4Gb, this is likely +to be the source of your problem, and you can modify it with: + +\footnotesize +\begin{verbatim} +mysql bacula +ALTER TABLE File MAX_ROWS=281474976710656; +\end{verbatim} +\normalsize + +Alternatively you can modify your /etc/my.conf file and in the +[mysqld] section set: + +\footnotesize +\begin{verbatim} +set-variable = myisam_data_pointer_size=6 +\end{verbatim} +\normalsize + +The above row and point size changes should already be the default on MySQL +version 5.x, so these changes should only be necessary on MySQL 4.x +depending on the size of your catalog database. + + +\label{RepairingPSQL} \subsection*{Repairing Your PostgreSQL Database} \index[general]{Database!Repairing Your PostgreSQL } \index[general]{Repairing Your PostgreSQL Database } @@ -186,9 +228,9 @@ The same considerations apply that are indicated above for MySQL. That is, consult the PostgreSQL documents for how to repair the database, and also consider using Bacula's dbcheck program if the conditions are reasonable for using (see above). -\label{CompactingPostgres} -\subsection*{Performance Issues} +\label{DatabasePerformance} +\subsection*{Database Performance Issues} \index[general]{Database Performance Issues} \index[general]{Performance!Database} \addcontentsline{toc}{subsection}{Database Performance Issues} @@ -216,7 +258,7 @@ give even further performance improvements at the risk of a database corruption if your system crashes. For PostgreSQL, you might want to consider turning fsync off. Of course -doing so can cause corrupted databases in the even of a machine crash. +doing so can cause corrupted databases in the event of a machine crash. There are many different ways that you can tune PostgreSQL, the following document discusses a few of them: \elink{ @@ -290,7 +332,38 @@ CREATE INDEX file_jpf_idx on File (Job, FilenameId, PathId); \end{verbatim} \normalsize -\subsubsection*{SQLit Indexes} +Though normally not a problem, you should ensure that the indexes +defined for Filename and Path are both set to 255 characters. Some users +reported performance problems when their indexes were set to 50 characters. +To check, do: + +\footnotesize +\begin{verbatim} +mysql bacula +show index from Filename; +show index from Path; +\end{verbatim} +\normalsize + +and what is important is that for Filename, you have an index with +Key_name "Name" and Sub_part "255". For Pth, you should have a Key_name +"Path" and Sub_part "255". If one or the other does not exist or the +Sub_part is less that 255, you can drop and recreate the appropriate +index with: + +\footnotesize +\begin{verbatim} +mysql bacula +DROP INDEX Path on Path; +CREATE INDEX Path on Path (Path(255); + +DROP INDEX Name on Filename; +CREATE INDEX Name on Filename (Name(255)); +\end{verbatim} +\normalsize + + +\subsubsection*{SQLite Indexes} On SQLite, you can check if you have the proper indexes by: \footnotesize @@ -307,12 +380,13 @@ create them with the following commands: \begin{verbatim} mysql bacula CREATE INDEX file_jobid_idx on File (JobId); -CREATE INDEX file_jpf_idx on File (Job, FilenameId, PathId); +CREATE INDEX file_jfp_idx on File (Job, FilenameId, PathId); \end{verbatim} \normalsize +\label{CompactingPostgres} \subsection*{Compacting Your PostgreSQL Database} \index[general]{Database!Compacting Your PostgreSQL } \index[general]{Compacting Your PostgreSQL Database } @@ -333,7 +407,7 @@ ASCII file (bacula.sql) then reload it by doing the following: \footnotesize \begin{verbatim} -pg_dump bacula > bacula.sql +pg_dump -c bacula > bacula.sql cat bacula.sql | psql bacula rm -f bacula.sql \end{verbatim} @@ -344,6 +418,20 @@ fair amount of disk space. For example, you can {\bf cd} to the location of the Bacula database (typically /usr/local/pgsql/data or possible /var/lib/pgsql/data) and check the size. +There are certain PostgreSQL users who do not recommend the above +procedure. They have the following to say: +PostgreSQL does not +need to be dumped/restored to keep the database efficient. A normal +process of vacuuming will prevent the database from every getting too +large. If you want to fine-tweak the database storage, commands such +as VACUUM FULL, REINDEX, and CLUSTER exist specifically to keep you +from having to do a dump/restore. + +Finally, you might want to look at the PostgreSQL documentation on +this subject at +\elink{http://www.postgresql.org/docs/8.1/interactive/maintenance.html} +{http://www.postgresql.org/docs/8.1/interactive/maintenance.html}. + \subsection*{Compacting Your SQLite Database} \index[general]{Compacting Your SQLite Database } \index[general]{Database!Compacting Your SQLite } diff --git a/docs/manual/filedconf.tex b/docs/manual/filedconf.tex index e18d9e21..0bc74bcb 100644 --- a/docs/manual/filedconf.tex +++ b/docs/manual/filedconf.tex @@ -244,14 +244,16 @@ allowed to connect to this Client. \index[fd]{Directive!Name} The name of the Director that may contact this Client. This name must be the same as the name specified on the Director resource in the Director's -configuration file. This record is required. +configuration file. Note, the case (upper/lower) of the characters in +the name are significant (i.e. S is not the same as s). This directive +is required. \item [Password = \lt{}password\gt{}] \index[fd]{Password} \index[fd]{Directive!Password} Specifies the password that must be supplied for a Director to be authorized. This password must be the same as the password specified in the Client -resource in the Director's configuration file. This record is required. +resource in the Director's configuration file. This directive is required. \item [Monitor = \lt{}yes|no\gt{}] \index[fd]{Monitor} diff --git a/docs/manual/pools.tex b/docs/manual/pools.tex index 8f6aaa97..6b0be004 100644 --- a/docs/manual/pools.tex +++ b/docs/manual/pools.tex @@ -3,9 +3,9 @@ \section*{Automated Disk Backup} \label{_ChapterStart11} -\index[general]{Volumes!Using Pools to Manage } +\index[general]{Volumes!Using Pools to Manage} \index[general]{Disk!Automated Backup} -\index[general]{Using Pools to Manage Volumes } +\index[general]{Using Pools to Manage Volumes} \index[general]{Automated Disk Backup} \addcontentsline{toc}{section}{Using Pools to Manage Volumes} \addcontentsline{toc}{section}{Automated Disk Backup} @@ -21,7 +21,7 @@ Volumes, but most of the information applies equally well to tape Volumes. \label{TheProblem} \subsection*{The Problem} -\index[general]{Problem } +\index[general]{Problem} \addcontentsline{toc}{subsection}{Problem} A site that I administer (a charitable organization) had a tape DDS-3 tape @@ -32,7 +32,7 @@ necessary cassettes was more expensive than their budget could handle. \label{TheSolution} \subsection*{The Solution} -\index[general]{Solution } +\index[general]{Solution} \addcontentsline{toc}{subsection}{Solution} They want to maintain 6 months of backup data, and be able to access the old @@ -50,14 +50,16 @@ same amount of data, and to have Bacula write to disk files. The rest of this chapter will explain how to setup Bacula so that it would automatically manage a set of disk files with the minimum intervention on my -part. The system has been running since 22 January 2004 until today (08 April -2004) with no intervention. Since we have not yet crossed the six month -boundary, we still lack some data to be sure the system performs as desired. -\label{OverallDesign} +part. The system has been running since 22 January 2004 until today (17 +September 2006) with no intervention, with the exception that I had to +add a second 120GB hard disk after a year because their needs grew +over that time to more than the 120GB (168GB to be exact). The only other +intervention I have made is a periodic (about once a year) Bacula upgrade. +\label{OverallDesign} \subsection*{Overall Design} -\index[general]{Overall Design } -\index[general]{Design!Overall } +\index[general]{Overall Design} +\index[general]{Design!Overall} \addcontentsline{toc}{subsection}{Overall Design} Getting Bacula to write to disk rather than tape in the simplest case is @@ -100,8 +102,8 @@ of volumes and a different Retention period to accomplish the requirements. \label{FullPool} \subsubsection*{Full Pool} -\index[general]{Pool!Full } -\index[general]{Full Pool } +\index[general]{Pool!Full} +\index[general]{Full Pool} \addcontentsline{toc}{subsubsection}{Full Pool} Putting a single Full backup on each Volume, will require six Full save @@ -131,8 +133,8 @@ manual from the start, but why not use the features of Bacula. \label{DiffPool} \subsubsection*{Differential Pool} -\index[general]{Pool!Differential } -\index[general]{Differential Pool } +\index[general]{Pool!Differential} +\index[general]{Differential Pool} \addcontentsline{toc}{subsubsection}{Differential Pool} For the Differential backup Pool, we choose a retention period of a bit longer @@ -161,8 +163,8 @@ the expense here is a few GB which is not too serious. \label{IncPool} \subsubsection*{Incremental Pool} -\index[general]{Incremental Pool } -\index[general]{Pool!Incremental } +\index[general]{Incremental Pool} +\index[general]{Pool!Incremental} \addcontentsline{toc}{subsubsection}{Incremental Pool} Finally, here is the resource for the Incremental Pool: @@ -192,8 +194,8 @@ doesn't exceed the disk capacity. \label{Example} \subsection*{The Actual Conf Files} -\index[general]{Files!Actual Conf } -\index[general]{Actual Conf Files } +\index[general]{Files!Actual Conf} +\index[general]{Actual Conf Files} \addcontentsline{toc}{subsection}{Actual Conf Files} The following example shows you the actual files used, with only a few minor @@ -210,7 +212,7 @@ Director { # define myself WorkingDirectory = "/home/bacula/working" PidDirectory = "/home/bacula/working" Maximum Concurrent Jobs = 1 - Password = " " + Password = " *** CHANGE ME ***" Messages = Standard } # By default, this job will back up to disk in /tmp @@ -229,46 +231,104 @@ Job { Write Bootstrap = "/home/bacula/working/client.bsr" Priority = 10 } + +# Backup the catalog database (after the nightly save) +Job { + Name = "BackupCatalog" + Type = Backup + Client = client-fd + FileSet="Catalog" + Schedule = "WeeklyCycleAfterBackup" + Storage = File + Messages = Standard + Pool = Default + # This creates an ASCII copy of the catalog + RunBeforeJob = "/home/bacula/bin/make_catalog_backup bacula bacula" + # This deletes the copy of the catalog + RunAfterJob = "/home/bacula/bin/delete_catalog_backup" + Write Bootstrap = "/home/bacula/working/BackupCatalog.bsr" + Priority = 11 # run after main backup +} + +# Standard Restore template, to be changed by Console program +Job { + Name = "RestoreFiles" + Type = Restore + Client = havana-fd + FileSet="Full Set" + Storage = File + Messages = Standard + Pool = Default + Where = /tmp/bacula-restores +} + + + # List of files to be backed up FileSet { Name = "Full Set" - Include = signature=SHA1 compression=GZIP9 { - / - /usr - /home + Include = { Options { signature=SHA1; compression=GZIP9 } + File = / + File = /usr + File = /home + File = /boot + File = /var + File = /opt } Exclude = { - /proc /tmp /.journal /.fsck + File = /proc + File = /tmp + File = /.journal + File = /.fsck + ... } } Schedule { Name = "WeeklyCycle" - Run = Full 1st sun at 1:05 - Run = Differential 2nd-5th sun at 1:05 - Run = Incremental mon-sat at 1:05 + Run = Full 1st sun at 2:05 + Run = Differential 2nd-5th sun at 2:05 + Run = Incremental mon-sat at 2:05 } + +# This schedule does the catalog. It starts after the WeeklyCycle +Schedule { + Name = "WeeklyCycleAfterBackup" + Run = Full sun-sat at 2:10 +} + +# This is the backup of the catalog +FileSet { + Name = "Catalog" + Include { Options { signature=MD5 } + File = /home/bacula/working/bacula.sql + } +} + Client { Name = client-fd Address = client FDPort = 9102 Catalog = MyCatalog - Password = " " + Password = " *** CHANGE ME ***" AutoPrune = yes # Prune expired Jobs/Files Job Retention = 6 months File Retention = 60 days } + Storage { Name = File Address = localhost SDPort = 9103 - Password = " " + Password = " *** CHANGE ME ***" Device = FileStorage Media Type = File } + Catalog { Name = MyCatalog dbname = bacula; user = bacula; password = "" } + Pool { Name = Full-Pool Pool Type = Backup @@ -277,28 +337,31 @@ Pool { Volume Retention = 6 months Maximum Volume Jobs = 1 Label Format = Full- - Maximum Volumes = 6 + Maximum Volumes = 8 } + Pool { Name = Inc-Pool Pool Type = Backup Recycle = yes # automatically recycle Volumes AutoPrune = yes # Prune expired volumes - Volume Retention = 20 days + Volume Retention = 14 days Maximum Volume Jobs = 6 Label Format = Inc- - Maximum Volumes = 5 + Maximum Volumes = 6 } + Pool { Name = Diff-Pool Pool Type = Backup Recycle = yes AutoPrune = yes - Volume Retention = 40 days + Volume Retention = 35 days Maximum Volume Jobs = 1 Label Format = Diff- - Maximum Volumes = 6 + Maximum Volumes = 10 } + Messages { Name = Standard mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\" @@ -325,7 +388,7 @@ Storage { # definition of myself } Director { Name = bacula-dir - Password = " " + Password = " *** CHANGE ME ***" } Device { Name = FileStorage diff --git a/docs/manual/progs.tex b/docs/manual/progs.tex index 3de01ffc..c0db6600 100644 --- a/docs/manual/progs.tex +++ b/docs/manual/progs.tex @@ -992,7 +992,6 @@ a LANG=''en\_US'' immediately before the bsmtp call. \index[general]{Dbcheck} \index[general]{program!dbcheck} \addcontentsline{toc}{subsection}{dbcheck} - {\bf dbcheck} is a simple program that will search for logical inconsistencies in the Bacula tables in your database, and optionally fix them. It is a database maintenance routine, in the sense that it can diff --git a/docs/manual/recycling.tex b/docs/manual/recycling.tex index 65e6c2cb..cb541fee 100644 --- a/docs/manual/recycling.tex +++ b/docs/manual/recycling.tex @@ -224,6 +224,7 @@ The default is 1 year. associated with a Job by adding {\bf Prune Files = yes} to the Job resource. \label{Recycling} +\label{RecyclingAlgorithm} \subsection*{Recycling Algorithm} \index[general]{Algorithm!Recycling } \index[general]{Recycling Algorithm } diff --git a/docs/manual/restore.tex b/docs/manual/restore.tex index 3799650c..4085e774 100644 --- a/docs/manual/restore.tex +++ b/docs/manual/restore.tex @@ -956,13 +956,21 @@ Bacula}{_ChapterRescue} chapter of this manual. \item[Solution] For SQLite, use the vacuum command to try to fix the database. For either MySQL or PostgreSQL, see the vendor's documentation. They have specific tools - that check and repair databases. + that check and repair databases, see the \ilink{database + repair}{DatabaseRepair} sections of this manual for links to vendor + information. Assuming the above does not resolve the problem, you will need to restore - or rebuild your catalog. + or rebuild your catalog. Note, if it is a matter of some + inconsistencies in the Bacula tables rather than a broken database, then + running \ilink{dbcheck}{dbcheck} might help, but you will need to ensure + that your database indexes are properly setup. Please see + the \ilink{Database Performance Issues}{DatabasePerformance} sections + of this manual for more details. + \item[Problem] How do I restore my catalog? -\item[Solution] +\item[Solution with a Catalog backup] If you have backed up your database nightly (as you should) and you have made a bootstrap file, you can immediately load back your database (or the ASCII SQL output). Make a copy of your current @@ -1015,7 +1023,7 @@ OK to run? (yes/mod/no): you will probably erase your newly restored database tables. -\item[Solution] +\item[Solution with a Job listing] If you did save your database but did not make a bootstrap file, then recovering the database is more difficult. You will probably need to use bextract to extract the backup copy. First you should locate the @@ -1107,6 +1115,16 @@ FileIndex=1-1 backup, you do NOT want to do a {\bf make_bacula_tables} command, or you will probably erase your newly restored database tables. +\item [Solution withou a Job Listing] + If you do not have a job listing, then it is a bit more difficult. + Either you use the \ilink{bscan}{bscan} program to scan the contents + of your tape into a database, which can be very time consuming + depending on the size of the tape, or you can use the \ilink{bls}{bls} + program to list everything on the tape, and reconstruct a bootstrap + file from the bls listing for the file or files you want following + the instructions given above. + + There is a specific example of how to use {\bf bls} below. \item [Problem] I try to restore the last known good full backup by specifying @@ -1174,7 +1192,8 @@ select VolumeName from JobMedia,Media where JobId=1 and JobMedia.MediaId=Media.M know the Volume to which it was backed up. \item [Solution] - Use {\bf bls} to indicate where it is on the tape. For example: + Either bscan the tape, or use {\bf bls} to indicate where it is on the + tape. For example: \footnotesize \begin{verbatim} diff --git a/docs/manual/version.tex b/docs/manual/version.tex index c1f4d251..a643c871 100644 --- a/docs/manual/version.tex +++ b/docs/manual/version.tex @@ -1 +1 @@ -1.39.23 (10 September 2006) +1.39.23 (16 September 2006) -- 2.39.5