From 4363c62609b0c3d8f6e9ec62ada90f6ee958d85b Mon Sep 17 00:00:00 2001 From: Kern Sibbald Date: Fri, 15 Dec 2006 20:11:16 +0000 Subject: [PATCH] Updates --- docs/README | 5 + docs/developers/version.tex | 2 +- docs/home-page/pages/support.php | 7 +- docs/manual-de/version.tex | 2 +- docs/manual/console.tex | 64 +- docs/manual/critical.tex | 2 +- docs/manual/dirdconf.tex | 16 + docs/manual/dvd.tex | 16 +- docs/manual/fileset.tex | 2 +- docs/manual/restore.tex | 147 ++-- docs/manual/storedconf.tex | 9 +- docs/manual/thanks.tex | 19 +- docs/manual/tutorial.tex | 2 +- docs/techlogs/2006/projects | 1229 ++++++++++++++++++++++++++++++ 14 files changed, 1420 insertions(+), 102 deletions(-) create mode 100644 docs/techlogs/2006/projects diff --git a/docs/README b/docs/README index f6d4f612..368e5410 100644 --- a/docs/README +++ b/docs/README @@ -12,6 +12,11 @@ the following subdirectories: Build instructions: +You must have at least the following packages loaded: +latex2html +te_latex +tetex + First, you must do: cd diff --git a/docs/developers/version.tex b/docs/developers/version.tex index e270280d..6f9c56e2 100644 --- a/docs/developers/version.tex +++ b/docs/developers/version.tex @@ -1 +1 @@ -1.39.29 (22 November 2006) +1.39.30 (08 December 2006) diff --git a/docs/home-page/pages/support.php b/docs/home-page/pages/support.php index af9108b9..5fa0812e 100644 --- a/docs/home-page/pages/support.php +++ b/docs/home-page/pages/support.php @@ -19,8 +19,11 @@

For support, send an email to bacula-users at lists.sourceforge.net, and if you are - specific enough, some kind Bacula user will help you. The - previous address was modified to prevent easy use by spammers. + specific enough, some kind Bacula user will help you. Please + note that if you don't at least specify what version of Bacula + and what platform you are using, it will not be easy to get a + valid answer. The email address noted above + was modified to prevent easy use by spammers. To use it, you must replace the at with an @ symbol. Due to the increasing volumes of spam on the list, you must be subscribed to it to be able to send and email to it. The diff --git a/docs/manual-de/version.tex b/docs/manual-de/version.tex index e270280d..6f9c56e2 100644 --- a/docs/manual-de/version.tex +++ b/docs/manual-de/version.tex @@ -1 +1 @@ -1.39.29 (22 November 2006) +1.39.30 (08 December 2006) diff --git a/docs/manual/console.tex b/docs/manual/console.tex index 2be74df0..0544c10a 100644 --- a/docs/manual/console.tex +++ b/docs/manual/console.tex @@ -285,19 +285,26 @@ The following commands are currently implemented: \item [{add [pool=\lt{}pool-name\gt{} storage=\lt{}storage\gt{} jobid=\lt{}JobId\gt{}]} ] \index[console]{add} - This command is not normally used since Bacula will normally - do it automatically as Volumes are labeled. However, if you need it, - this command is used to add Volumes to an existing Pool. The Volume names - entered are placed in the Catalog and thus become available for backup - operations. Normally, the {\bf label} command is used rather than this - command because the {\bf label} command labels the physical media (tape) and - does the equivalent of the {\bf add} command. This command affects only the - Catalog and not the physical media (data on Volumes). The physical media must - exist and be labeled before use (usually with the {\bf label} command). This - command can, however, be useful if you wish to add a number of Volumes to the - Pool that will be physically labeled at a later time. It can also be useful - if you are importing a tape from another site. Please see the {\bf label} - command below for the list of legal characters in a Volume name. + This command is used to add Volumes to an existing Pool. That is, + it creates the Volume name in the catalog and inserts into the Pool + in the catalog, but does not attempt to access the physical Volume. + Once + added, Bacula expects that Volume to exist and to be labeled. + This command is not normally used since Bacula will + automatically do the equivalent when Volumes are labeled. However, + there may be times when you have removed a Volume from the catalog + and want to later add it back. + + Normally, the {\bf label} command is used rather than this command + because the {\bf label} command labels the physical media (tape) and + does the equivalent of the {\bf add} command. The {\bf add} command + affects only the Catalog and not the physical media (data on Volumes). + The physical media must exist and be labeled before use (usually with + the {\bf label} command). This command can, however, be useful if you + wish to add a number of Volumes to the Pool that will be physically + labeled at a later time. It can also be useful if you are importing a + tape from another site. Please see the {\bf label} command below for + the list of legal characters in a Volume name. \item [autodisplay on/off] \index[console]{autodisplay on/off} @@ -404,7 +411,7 @@ delete Job JobId=n,m,o-r,t ... {\bf Enabled} directive or using the console {\bf enable} command. The next time the Director is restarted or the conf file is reloaded, the Enable/Disable state will be set to the value in the Job resource - (default enabled). + (default enabled) as defined in the bacula-dir.conf file. \item [enable job\lt{}job-name\gt{}] \index[console]{enable} @@ -413,7 +420,7 @@ delete Job JobId=n,m,o-r,t ... {\bf Enabled} directive or using the console {\bf disable} command. The next time the Director is restarted or the conf file is reloaded, the Enable/Disable state will be set to the value in the Job resource - (default enabled). + (default enabled) as defined in the bacula-dir.conf file. \label{estimate} \item [estimate] @@ -499,7 +506,7 @@ label storage=\lt{}storage-name\gt{} volume=\lt{}volume-name\gt{} operator errors. Please note, when labeling a blank tape, Bacula will get {\bf read I/O - error} when it attempts to ensure that the tape is already labeled. If + error} when it attempts to ensure that the tape is not already labeled. If you wish to avoid getting these messages, please write an EOF mark on your tape before attempting to label it: @@ -542,16 +549,15 @@ The preferable method to relabel a tape is to first {\bf purge} the volume, either automatically, or explicitly with the {\bf purge} command, then use the {\bf relabel} command described below. -If your autochanger has barcode labels, you can label all the Volumes in your -autochanger one after another by using the {\bf label barcodes} command. For -each tape in the changer containing a barcode, Bacula will mount the tape and -then label it with the same name as the barcode. An appropriate Media record -will also be created in the catalog. Any barcode that begins with the same -characters as specified on the "CleaningPrefix=xxx" directive in the -Director's Pool resource, will be -treated as a cleaning tape, and will not be labeled. However, -an entry for the cleaning tape will be created in -the catalog. For example with: +If your autochanger has barcode labels, you can label all the Volumes in +your autochanger one after another by using the {\bf label barcodes} +command. For each tape in the changer containing a barcode, Bacula will +mount the tape and then label it with the same name as the barcode. An +appropriate Media record will also be created in the catalog. Any barcode +that begins with the same characters as specified on the +"CleaningPrefix=xxx" directive in the Director's Pool resource, will be +treated as a cleaning tape, and will not be labeled. However, an entry for +the cleaning tape will be created in the catalog. For example with: \footnotesize \begin{verbatim} @@ -563,8 +569,8 @@ the catalog. For example with: \end{verbatim} \normalsize -Any slot containing a barcode of CLNxxxx will be treated as a cleaning tape -and will not be mounted. Note, the full form of the command is: +Any slot containing a barcode of CLNxxxx will be treated as a cleaning tape +and will not be mounted. Note, the full form of the command is: \footnotesize \begin{verbatim} @@ -887,7 +893,7 @@ restore storage=\lt{}storage-name\gt{} client=\lt{}client-name\gt{} specified, you will be prompted. The {\bf all} specification tells the restore command to restore all files. If it is not specified, you will be prompted for the files to restore. For details of the {\bf restore} - command, please see the \ilink{Restore Chapter}{_ChapterStart13} of this + command, please see the \ilink{Restore Chapter}{RestoreChapter} of this manual. \item [run] diff --git a/docs/manual/critical.tex b/docs/manual/critical.tex index 14dddf47..d92d1637 100644 --- a/docs/manual/critical.tex +++ b/docs/manual/critical.tex @@ -53,7 +53,7 @@ production, use the checklist anyway). \item Do at least one restore of files. If you backup multiple OS types (Linux, Solaris, HP, MacOS, FreeBSD, Win32, ...), restore files from each system type. The - \ilink{Restoring Files}{_ChapterStart13} chapter shows you how. + \ilink{Restoring Files}{RestoreChapter} chapter shows you how. \item Write a bootstrap file to a separate system for each backup job. The Write Bootstrap directive is described in the \ilink{Director Configuration}{writebootstrap} chapter of the diff --git a/docs/manual/dirdconf.tex b/docs/manual/dirdconf.tex index 617c6711..b920ef22 100644 --- a/docs/manual/dirdconf.tex +++ b/docs/manual/dirdconf.tex @@ -310,6 +310,14 @@ Only a single type ({\bf Backup}, {\bf Restore}, ...) can be specified for any job. If you want to backup multiple FileSets on the same Client or multiple Clients, you must define a Job for each one. +Note, you define only a single Job to do the Full, Differential, and +Incremental backups since the different backup levels are tied together by +a unique Job name. Normally, you will have only one Job per Client, but +if a client has a really huge number of files (more than several million), +you might want to split it into to Jobs each with a different FileSet +covering only part of the total files. + + \begin{description} \item [Job] @@ -1243,6 +1251,14 @@ RunScript { a prior Full save has failed, you wish the very next backup to be a Full save rather than whatever level it is started as. + There are several points that must be taken into account when using this + directive: first, a failed job is defined as one that has not terminated + normally, which includes any running job of the same name (you need to + ensure that two jobs of the same name do not run simultaneously); + secondly, the {\bf Ignore FileSet Changes} directive is not considered + when checing for failed levels, which means that any FileSet change will + trigger a rerun. + \item [Spool Data = \lt{}yes|no\gt{}] \index[dir]{Spool Data} \index[dir]{Directive!Spool Data} diff --git a/docs/manual/dvd.tex b/docs/manual/dvd.tex index c2a72060..c9725e5d 100644 --- a/docs/manual/dvd.tex +++ b/docs/manual/dvd.tex @@ -34,7 +34,7 @@ The fact that Bacula cannot use the OS to write directly to the DVD makes the whole process a bit more error prone than writing to a disk or a tape, but nevertheless, it does work if you use some care to set it up properly. However, at the current time -(26 August 2006) we still consider this code to be experimental and of +(version 1.39.30 -- 12 December 2006) we still consider this code to be BETA quality. As a consequence, please do careful testing before relying on DVD backups in production. @@ -225,6 +225,20 @@ The following directives are added to the Director's Job resource. \addcontentsline{toc}{subsection}{Other Points} \begin{itemize} +\item Please be sure that you have any automatic DVD mounting + disabled before running Bacula -- this includes auto mounting + in /etc/fstab, hotplug, ... If the DVD is automatically + mounted by the OS, it will cause problems when Bacula tries + to mount/unmount the DVD. +\item Please be sure that you the directive {\bf Write Part After Job} + set to {\bf yes}, otherwise the last part of the data to be + written will be left in the DVD spool file and not written to + the DVD. The DVD will then be unreadable until this last part + is written. If you have a series of jobs that are run one at + a time, you can turn this off until the last job is run. +\item The current code is not designed to have multiple simultaneous + jobs writing to the DVD. As a consequence, please ensure that + only one DVD backup job runs at any time. \item Writing and reading of DVD+RW seems to work quite reliably provided you are using the patched dvd+rw-mediainfo programs. On the other hand, we do not have enough information to ensure diff --git a/docs/manual/fileset.tex b/docs/manual/fileset.tex index a243a7a4..e45e0d16 100644 --- a/docs/manual/fileset.tex +++ b/docs/manual/fileset.tex @@ -51,7 +51,7 @@ defined for each Backup job. \index[dir]{Directive!Enable VSS} If this directive is set to {\bf yes} the File daemon will be notified that the user wants to use a Volume Shadow Copy Service (VSS) backup - for this job. The default is {\bf no}. This directive is effective + for this job. The default is {\bf yes}. This directive is effective only for VSS enabled Win32 File daemons. It permits a consistent copy of open files to be made for cooperating writer applications, and for applications that are not VSS away, Bacula can at least copy open files. diff --git a/docs/manual/restore.tex b/docs/manual/restore.tex index a807f010..40227e6a 100644 --- a/docs/manual/restore.tex +++ b/docs/manual/restore.tex @@ -2,7 +2,7 @@ %% \section*{The Bacula Console Restore Command} -\label{_ChapterStart13} +\label{RestoreChapter} \index[general]{Command!Bacula Console Restore } \index[general]{Bacula Console Restore Command } \addcontentsline{toc}{section}{Bacula Console Restore Command} @@ -63,8 +63,9 @@ interactively walk up and down the file tree selecting individual files to be restored. This mode is somewhat similar to the standard Unix {\bf restore} program's interactive file selection mode. -If your Files have been pruned, the {\bf restore} command will be unable -to find any files to restore. See below for more details on this. +If a Job's file records have been pruned from the catalog, the {\bf +restore} command will be unable to find any files to restore. See below +for more details on this. Within the Console program, after entering the {\bf restore} command, you are presented with the following selection prompt: @@ -92,6 +93,10 @@ Select item: (1-12): \end{verbatim} \normalsize +There are a lot of options, and as a point of reference, most people will +want to slect item 5 (the most recent backup for a client). The details +of the above options are: + \begin{itemize} \item Item 1 will list the last 20 jobs run. If you find the Job you want, you can then select item 3 and enter its JobId(s). @@ -101,20 +106,27 @@ Select item: (1-12): \item Item 3 allows you the enter a list of comma separated JobIds whose files will be put into the directory tree. You may then select which - files from those JobIds to restore. - -\item Item 4 allows you to enter any arbitrary SQL command. This is probably - the most primitive way of finding the desired JobIds, but at the same time, - the most flexible. Once you have found the JobId(s), you can select item 3 - and enter them. - -\item Item 5 will automatically select the most recent Full backup and all + files from those JobIds to restore. Normally, you would use this option + if you have a particular version of a file that you want to restore and + you know its JobId. The most common options (5 and 6) will not select + a job that did not terminate normally, so if you know a file is + backed up by a Job that failed (possibly because of a system crash), you + can access it through this option by specifying the JobId. + +\item Item 4 allows you to enter any arbitrary SQL command. This is + probably the most primitive way of finding the desired JobIds, but at + the same time, the most flexible. Once you have found the JobId(s), you + can select item 3 and enter them. + +\item Item 5 will automatically select the most recent Full backup and all subsequent incremental and differential backups for a specified Client. These are the Jobs and Files which, if reloaded, will restore your system to the most current saved state. It automatically enters the - JobIds found into the directory tree. This is probably the most - convenient of all the above options to use if you wish to restore a - selected Client to its most recent state. + JobIds found into the directory tree in an optimal way such that only + the most recent copy of any particular file found in the set of Jobs + will be restored. This is probably the most convenient of all the above + options to use if you wish to restore a selected Client to its most + recent state. There are two important things to note. First, this automatic selection will never select a job that failed (terminated with an error status). @@ -134,18 +146,24 @@ Select item: (1-12): will then propose doing a full restore (non-selective) of those JobIds. This is possible because Bacula still knows where the beginning of the Job data is on the Volumes, even if it does not know where particular - files are located. + files are located or what their names are. \item Item 6 allows you to specify a date and time, after which Bacula will automatically select the most recent Full backup and all subsequent incremental and differential backups that started before the specified date - and time. + and time. \item Item 7 allows you to specify one or more filenames (complete path required) to be restored. Each filename is entered one at a time or if you prefix a filename with the less-than symbol (\lt{}) Bacula will read that - file and assume it is a list of filenames to be restored. The filename entry - mode is terminated by entering a blank line. + file and assume it is a list of filenames to be restored. If you + prefix the filename with a question mark (?), then the filename will + be interpreted as an SQL table name, and Bacula will include the rows + of that table in the list to be restored. The table must contain the + JobId in the first column and the FileIndex in the second column. + This table feature is intended for external programs that want to build + their own list of files to be restored. + The filename entry mode is terminated by entering a blank line. \item Item 8 allows you to specify a date and time before entering the filenames. See Item 7 above for more details. @@ -175,7 +193,8 @@ Select item: (1-12): \end{itemize} As an example, suppose that we select item 5 (restore to most recent state). -It will then ask for the desired Client, which on my system, will print all +If you have not specified a client=xxx on the command line, it +it will then ask for the desired Client, which on my system, will print all the Clients found in the database as follows: \footnotesize @@ -191,14 +210,13 @@ Defined clients: 8: RufusVerify 9: Watchdog Select Client (File daemon) resource (1-9): - \end{verbatim} \normalsize -You will probably have far fewer Clients than this example, and if you have -only one Client, it will be automatically selected. In this case, I enter +You will probably have far fewer Clients than this example, and if you have +only one Client, it will be automatically selected. In this case, I enter {\bf Rufus} to select the Client. Then Bacula needs to know what FileSet is -to be restored, so it prompts with: +to be restored, so it prompts with: \footnotesize \begin{verbatim} @@ -206,13 +224,13 @@ The defined FileSet resources are: 1: Full Set 2: Kerns Files Select FileSet resource (1-2): - \end{verbatim} \normalsize -I choose item 1, which is my full backup. Normally, you will only have a -single FileSet for each Job, and if your machines are similar (all Linux) you -may only have one FileSet for all your Clients. +If you have only one FileSet defined for the Client, it will be selected +automatically. I choose item 1, which is my full backup. Normally, you +will only have a single FileSet for each Job, and if your machines are +similar (all Linux) you may only have one FileSet for all your Clients. At this point, {\bf Bacula} has all the information it needs to find the most recent set of backups. It will then query the database, which may take a bit @@ -266,7 +284,8 @@ prompts with the dollar sign (\$) to indicate that you may enter commands to move around the directory tree and to select files. If you want all the files to automatically be marked when the directory -tree is built, enter the command {\bf restore all}. +tree is built, you could have entered the command {\bf restore all}, or +at the \$ prompt, you can simply enter {\bf mark *}. Instead of choosing item 5 on the first menu (Select the most recent backup for a client), if we had chosen item 3 (Enter list of JobIds to select) and we @@ -341,10 +360,13 @@ prints: \footnotesize \begin{verbatim} Bootstrap records written to /home/kern/bacula/working/restore.bsr -The restore job will require the following Volumes: - - DLT-19Jul02 - DLT-04Aug02 +The job will require the following + Volume(s) Storage(s) SD Device(s) +=========================================================================== + + DLT-19Jul02 Tape DLT8000 + DLT-04Aug02 Tape DLT8000 + 128401 files selected to restore. Run Restore job JobName: kernsrestore @@ -353,8 +375,10 @@ Where: /tmp/bacula-restores Replace: always FileSet: Kerns Files Client: Rufus -Storage: SDT-10000 -JobId: *None* +Storage: Tape +When: 2006-12-11 18:20:33 +Catalog: MyCatalog +Priority: 10 OK to run? (yes/mod/no): \end{verbatim} @@ -551,24 +575,24 @@ The full list of possible command line arguments are: \item {\bf before=YYYY-MM-DD HH:MM:SS} -- specify a date and time to which the system should be restored. Only Jobs started before the specified date/time will be selected, and as is the case for {\bf current} Bacula will -automatically find the most recent prior Full save and all Differential and -Incremental saves run before the date you specify. Note, this command is not -too user friendly in that you must specify the date/time exactly as shown. + automatically find the most recent prior Full save and all Differential and + Incremental saves run before the date you specify. Note, this command is not + too user friendly in that you must specify the date/time exactly as shown. \item {\bf file=filename} -- specify a filename to be restored. You must specify the full path and filename. Prefixing the entry with a less-than -sign + sign (\lt{}) will cause Bacula to assume that the filename is on your system and -contains a list of files to be restored. Bacula will thus read the list from -that file. Multiple file=xxx specifications may be specified on the command -line. + contains a list of files to be restored. Bacula will thus read the list from + that file. Multiple file=xxx specifications may be specified on the command + line. \item {\bf jobid=nnn} -- specify a JobId to be restored. \item {\bf pool=pool-name} -- specify a Pool name to be used for selection of Volumes when specifying options 5 and 6 (restore current system, and restore current system before given date). This permits you to have several Pools, -possibly one offsite, and to select the Pool to be used for restoring. + possibly one offsite, and to select the Pool to be used for restoring. \item {\bf yes} -- automatically run the restore without prompting for modifications (most useful in batch scripts). - \end{itemize} +\end{itemize} \subsection*{Restoring Directory Attributes} \index[general]{Attributes!Restoring Directory } @@ -717,6 +741,9 @@ what it is now after each individual test: \item Set "Minimum Block Size = 512" and "Maximum Block Size = 512" and try the restore. If you are able to determine the block size your drive was previously using, you should try that size if 512 does not work. + This is a really horrible solution, and it is not at all recommended + to continue backing up your data without correcting this condition. + Please see the Tape Testing chapter for more on this. \item Try editing the restore.bsr file at the Run xxx yes/mod/no prompt before starting the restore job and remove all the VolBlock statements. These are what causes Bacula to reposition the tape, and where problems @@ -1241,22 +1268,24 @@ Volume "DLT-22Apr05" \begin{verbatim} *query Available queries: - 1: List Job totals: - 2: List up to 20 places where a File is saved regardless of the directory: - 3: List where the most recent copies of a file are saved: - 4: List last 20 Full Backups for a Client: - 5: List all backups for a Client after a specified time - 6: List all backups for a Client - 7: List Volume Attributes for a selected Volume: - 8: List Volumes used by selected JobId: - 9: List Volumes to Restore All Files: - 10: List Pool Attributes for a selected Pool: - 11: List total files/bytes by Job: - 12: List total files/bytes by Volume: - 13: List Files for a selected JobId: - 14: List Jobs stored in a selected MediaId: - 15: List Jobs stored for a given Volume name: -Choose a query (1-15): + 1: List up to 20 places where a File is saved regardless of the +directory + 2: List where the most recent copies of a file are saved + 3: List last 20 Full Backups for a Client + 4: List all backups for a Client after a specified time + 5: List all backups for a Client + 6: List Volume Attributes for a selected Volume + 7: List Volumes used by selected JobId + 8: List Volumes to Restore All Files + 9: List Pool Attributes for a selected Pool + 10: List total files/bytes by Job + 11: List total files/bytes by Volume + 12: List Files for a selected JobId + 13: List Jobs stored on a selected MediaId + 14: List Jobs stored for a given Volume name + 15: List Volumes Bacula thinks are in changer + 16: List Volumes likely to need replacement from age or errors +Choose a query (1-16): \end{verbatim} \normalsize diff --git a/docs/manual/storedconf.tex b/docs/manual/storedconf.tex index e26db353..e116acbe 100644 --- a/docs/manual/storedconf.tex +++ b/docs/manual/storedconf.tex @@ -537,11 +537,12 @@ bacula-sd Alert: TapeAlert[32]: Interface: Problem with SCSI interface \index[sd]{Directive!Always Open} If {\bf Yes} (default), Bacula will always keep the device open unless specifically {\bf unmounted} by the Console program. This permits - Bacula to ensure that the tape drive is always available. If you set + Bacula to ensure that the tape drive is always available, and properly + positioned. If you set {\bf AlwaysOpen} to {\bf no} {\bf Bacula} will only open the drive when necessary, and at the end of the Job if no other Jobs are using the drive, it will be freed. The next time Bacula wants to append to a tape - on a drive that was freed, Bacula must rewind the tape and position to + on a drive that was freed, Bacula will rewind the tape and position it to the end. To avoid unnecessary tape positioning and to minimize unnecessary operator intervention, it is highly recommended that {\bf Always Open = yes}. This also ensures that the drive is available when @@ -558,7 +559,9 @@ bacula-sd Alert: TapeAlert[32]: Interface: Problem with SCSI interface Please note that if you set this directive to {\bf No} Bacula will release the tape drive between each job, and thus the next job will rewind the tape and position it to the end of the data. This can be a very time consuming - operation. + operation. In addition, with this directive set to no, certain multiple + drive autochanger operations will fail. We strongly recommend to keep + {\bf Always Open} set to {\bf Yes} \item [Volume Poll Interval = {\it time}] \index[sd]{Volume Poll Interval} diff --git a/docs/manual/thanks.tex b/docs/manual/thanks.tex index 5e35d700..34b2cdd1 100644 --- a/docs/manual/thanks.tex +++ b/docs/manual/thanks.tex @@ -5,6 +5,12 @@ \label{_ChapterStart44} \index[general]{Thanks } \addcontentsline{toc}{section}{Thanks} +I thank everyone who has helped this project. Unfortunately, I cannot +thank everyone (bad memory). However, the AUTHORS file in the main source +code directory should include the names of all persons who have contributed +to the Bacula project. Just the same, I would like to include thanks below +to special contributors as well as to the major contributors to the current +release. Thanks to Richard Stallman for starting the Free Software movement and for bringing us gcc and all the other GNU tools as well as the GPL license. @@ -44,20 +50,24 @@ different platforms for Bacula. Thanks to Christopher Hull for developing the native Win32 Bacula emulation code and for contributing it to the Bacula project. +Thanks to Robert Nelson for bringing our Win32 implementation up to par +with all the same features that exist in the Unix/Linux versions. In +addition, he has ported the Director and Storage daemon to Win32! + Thanks to Thorsten Engel for his excellent knowledge of Win32 systems, and for making the Win32 File daemon Unicode compatible, as well as making the Win32 File daemon interface to Microsoft's Volume Shadow Copy (VSS). These two are big pluses for Bacula! +Thanks to Landon Fuller for writing both the communications and the +data encryption code for Bacula. + Thanks to Arno Lehmann for his excellent and infatigable help and advice to users. Thanks to all the Bacula users, especially those of you who have contributed ideas, bug reports, patches, and new features. -Thanks to Nicolas Boichat for writing wx-console and the bacula-tray-monitor. -These are very nice GUI additions to Bacula. - The original variable expansion code used in the LabelFormat comes from the Open Source Software Project (www.ossp.org). It has been adapted and extended for use in Bacula. This code is now deprecated. @@ -68,6 +78,9 @@ release file contains a list of contributors. For all those who I have left out, please send me a reminder, and in any case, thanks for your contribution. +Thanks to the Free Software Foundation Europe e.V. for assuming the +responsibilities of protecting the Bacula copyright. + \subsection*{Copyrights and Trademarks} \index[general]{Trademarks!Copyrights and } \index[general]{Copyrights and Trademarks } diff --git a/docs/manual/tutorial.tex b/docs/manual/tutorial.tex index d209119d..fc396629 100644 --- a/docs/manual/tutorial.tex +++ b/docs/manual/tutorial.tex @@ -636,7 +636,7 @@ and view what files will be restored. For example, if I enter {\bf cd /home/kern/bacula/bacula-1.30} and then enter {\bf dir} I will get a listing of all the files in the Bacula source directory. On your system, the path will be somewhat different. For more information on this, please refer to the -\ilink{Restore Command Chapter}{_ChapterStart13} of this manual for +\ilink{Restore Command Chapter}{RestoreChapter} of this manual for more details. To exit this mode, simply enter: diff --git a/docs/techlogs/2006/projects b/docs/techlogs/2006/projects new file mode 100644 index 00000000..8234373b --- /dev/null +++ b/docs/techlogs/2006/projects @@ -0,0 +1,1229 @@ + +Projects: + Bacula Projects Roadmap + Prioritized by user vote 07 December 2005 + Status updated 15 December 2006 + +Summary: +Item 1: Implement data encryption (as opposed to comm encryption) +Item 2: Implement Migration that moves Jobs from one Pool to another. +Item 3: Accurate restoration of renamed/deleted files from +Item 4: Implement a Bacula GUI/management tool using Python. +Item 5: Implement Base jobs. +Item 6: Allow FD to initiate a backup +Item 7: Improve Bacula's tape and drive usage and cleaning management. +Item 8: Implement creation and maintenance of copy pools +Item 9: Implement new {Client}Run{Before|After}Job feature. +Item 10: Merge multiple backups (Synthetic Backup or Consolidation). +Item 11: Deletion of Disk-Based Bacula Volumes +Item 12: Directive/mode to backup only file changes, not entire file +Item 13: Multiple threads in file daemon for the same job +Item 14: Implement red/black binary tree routines. +Item 15: Add support for FileSets in user directories CACHEDIR.TAG +Item 16: Implement extraction of Win32 BackupWrite data. +Item 17: Implement a Python interface to the Bacula catalog. +Item 18: Archival (removal) of User Files to Tape +Item 19: Add Plug-ins to the FileSet Include statements. +Item 20: Implement more Python events in Bacula. +Item 21: Quick release of FD-SD connection after backup. +Item 22: Permit multiple Media Types in an Autochanger +Item 23: Allow different autochanger definitions for one autochanger. +Item 24: Automatic disabling of devices +Item 25: Implement huge exclude list support using hashing. + +Items complete and to be released in version 1.40.0: +Item 1: Implement data encryption (as opposed to comm encryption) +Item 2: Implement Migration that moves Jobs from one Pool to another. +Item 9: Implement new {Client}Run{Before|After}Job feature. +Item 16: Implement extraction of Win32 BackupWrite data. + +Items implemented but not tested and hence consequences are unknown: +Item 22: Permit multiple Media Types in an Autochanger + + +Below, you will find more information on future projects: + +Item 1: Implement data encryption (as opposed to comm encryption) + Date: 28 October 2005 + Origin: Sponsored by Landon and 13 contributors to EFF. + Status: Done: Landon Fuller has implemented this in 1.39.x. + + What: Currently the data that is stored on the Volume is not + encrypted. For confidentiality, encryption of data at + the File daemon level is essential. + Data encryption encrypts the data in the File daemon and + decrypts the data in the File daemon during a restore. + + Why: Large sites require this. + +Item 2: Implement Migration that moves Jobs from one Pool to another. + Origin: Sponsored by Riege Software International GmbH. Contact: + Daniel Holtkamp + Date: 28 October 2005 + Status: Done. Completed in version 1.39.31 by Kern. + + What: The ability to copy, move, or archive data that is on a + device to another device is very important. + + Why: An ISP might want to backup to disk, but after 30 days + migrate the data to tape backup and delete it from + disk. Bacula should be able to handle this + automatically. It needs to know what was put where, + and when, and what to migrate -- it is a bit like + retention periods. Doing so would allow space to be + freed up for current backups while maintaining older + data on tape drives. + + Notes: Riege Software have asked for the following migration + triggers: + Age of Job + Highwater mark (stopped by Lowwater mark?) + + Notes: Migration could be additionally triggered by: + Number of Jobs + Number of Volumes + +Item 3: Accurate restoration of renamed/deleted files from + Incremental/Differential backups + Date: 28 November 2005 + Origin: Martin Simmons (martin at lispworks dot com) + Status: + + What: When restoring a fileset for a specified date (including "most + recent"), Bacula should give you exactly the files and directories + that existed at the time of the last backup prior to that date. + + Currently this only works if the last backup was a Full backup. + When the last backup was Incremental/Differential, files and + directories that have been renamed or deleted since the last Full + backup are not currently restored correctly. Ditto for files with + extra/fewer hard links than at the time of the last Full backup. + + Why: Incremental/Differential would be much more useful if this worked. + + Notes: Item 14 (Merging of multiple backups into a single one) seems to + rely on this working, otherwise the merged backups will not be + truly equivalent to a Full backup. + + Kern: notes shortened. This can be done without the need for + inodes. It is essentially the same as the current Verify job, + but one additional database record must be written, which does + not need any database change. + + Kern: see if we can correct restoration of directories if + replace=ifnewer is set. Currently, if the directory does not + exist, a "dummy" directory is created, then when all the files + are updated, the dummy directory is newer so the real values + are not updated. + +Item 4: Implement a Bacula GUI/management tool using Python. + Origin: Kern + Date: 28 October 2005 + Status: Lucus is working on this for Python GTK+. + + What: Implement a Bacula console, and management tools + using Python and Qt or GTK. + + Why: Don't we already have a wxWidgets GUI? Yes, but + it is written in C++ and changes to the user interface + must be hand tailored using C++ code. By developing + the user interface using Qt designer, the interface + can be very easily updated and most of the new Python + code will be automatically created. The user interface + changes become very simple, and only the new features + must be implement. In addition, the code will be in + Python, which will give many more users easy (or easier) + access to making additions or modifications. + + Notes: This is currently being implemented using Python-GTK by + Lucas Di Pentima + +Item 5: Implement Base jobs. + Date: 28 October 2005 + Origin: Kern + Status: + + What: A base job is sort of like a Full save except that you + will want the FileSet to contain only files that are + unlikely to change in the future (i.e. a snapshot of + most of your system after installing it). After the + base job has been run, when you are doing a Full save, + you specify one or more Base jobs to be used. All + files that have been backed up in the Base job/jobs but + not modified will then be excluded from the backup. + During a restore, the Base jobs will be automatically + pulled in where necessary. + + Why: This is something none of the competition does, as far as + we know (except perhaps BackupPC, which is a Perl program that + saves to disk only). It is big win for the user, it + makes Bacula stand out as offering a unique + optimization that immediately saves time and money. + Basically, imagine that you have 100 nearly identical + Windows or Linux machine containing the OS and user + files. Now for the OS part, a Base job will be backed + up once, and rather than making 100 copies of the OS, + there will be only one. If one or more of the systems + have some files updated, no problem, they will be + automatically restored. + + Notes: Huge savings in tape usage even for a single machine. + Will require more resources because the DIR must send + FD a list of files/attribs, and the FD must search the + list and compare it for each file to be saved. + +Item 6: Allow FD to initiate a backup + Origin: Frank Volf (frank at deze dot org) + Date: 17 November 2005 + Status: + + What: Provide some means, possibly by a restricted console that + allows a FD to initiate a backup, and that uses the connection + established by the FD to the Director for the backup so that + a Director that is firewalled can do the backup. + + Why: Makes backup of laptops much easier. + +Item 7: Improve Bacula's tape and drive usage and cleaning management. + Date: 8 November 2005, November 11, 2005 + Origin: Adam Thornton , + Arno Lehmann + Status: + + What: Make Bacula manage tape life cycle information, tape reuse + times and drive cleaning cycles. + + Why: All three parts of this project are important when operating + backups. + We need to know which tapes need replacement, and we need to + make sure the drives are cleaned when necessary. While many + tape libraries and even autoloaders can handle all this + automatically, support by Bacula can be helpful for smaller + (older) libraries and single drives. Limiting the number of + times a tape is used might prevent tape errors when using + tapes until the drives can't read it any more. Also, checking + drive status during operation can prevent some failures (as I + [Arno] had to learn the hard way...) + + Notes: First, Bacula could (and even does, to some limited extent) + record tape and drive usage. For tapes, the number of mounts, + the amount of data, and the time the tape has actually been + running could be recorded. Data fields for Read and Write + time and Number of mounts already exist in the catalog (I'm + not sure if VolBytes is the sum of all bytes ever written to + that volume by Bacula). This information can be important + when determining which media to replace. The ability to mark + Volumes as "used up" after a given number of write cycles + should also be implemented so that a tape is never actually + worn out. For the tape drives known to Bacula, similar + information is interesting to determine the device status and + expected life time: Time it's been Reading and Writing, number + of tape Loads / Unloads / Errors. This information is not yet + recorded as far as I [Arno] know. A new volume status would + be necessary for the new state, like "Used up" or "Worn out". + Volumes with this state could be used for restores, but not + for writing. These volumes should be migrated first (assuming + migration is implemented) and, once they are no longer needed, + could be moved to a Trash pool. + + The next step would be to implement a drive cleaning setup. + Bacula already has knowledge about cleaning tapes. Once it + has some information about cleaning cycles (measured in drive + run time, number of tapes used, or calender days, for example) + it can automatically execute tape cleaning (with an + autochanger, obviously) or ask for operator assistance loading + a cleaning tape. + + The final step would be to implement TAPEALERT checks not only + when changing tapes and only sending the information to the + administrator, but rather checking after each tape error, + checking on a regular basis (for example after each tape + file), and also before unloading and after loading a new tape. + Then, depending on the drives TAPEALERT state and the known + drive cleaning state Bacula could automatically schedule later + cleaning, clean immediately, or inform the operator. + + Implementing this would perhaps require another catalog change + and perhaps major changes in SD code and the DIR-SD protocol, + so I'd only consider this worth implementing if it would + actually be used or even needed by many people. + + Implementation of these projects could happen in three distinct + sub-projects: Measuring Tape and Drive usage, retiring + volumes, and handling drive cleaning and TAPEALERTs. + +Item 8: Implement creation and maintenance of copy pools + Date: 27 November 2005 + Origin: David Boyes (dboyes at sinenomine dot net) + Status: + + What: I would like Bacula to have the capability to write copies + of backed-up data on multiple physical volumes selected + from different pools without transferring the data + multiple times, and to accept any of the copy volumes + as valid for restore. + + Why: In many cases, businesses are required to keep offsite + copies of backup volumes, or just wish for simple + protection against a human operator dropping a storage + volume and damaging it. The ability to generate multiple + volumes in the course of a single backup job allows + customers to simple check out one copy and send it + offsite, marking it as out of changer or otherwise + unavailable. Currently, the library and magazine + management capability in Bacula does not make this process + simple. + + Restores would use the copy of the data on the first + available volume, in order of copy pool chain definition. + + This is also a major scalability issue -- as the number of + clients increases beyond several thousand, and the volume + of data increases, transferring the data multiple times to + produce additional copies of the backups will become + physically impossible due to transfer speed + issues. Generating multiple copies at server side will + become the only practical option. + + How: I suspect that this will require adding a multiplexing + SD that appears to be a SD to a specific FD, but 1-n FDs + to the specific back end SDs managing the primary and copy + pools. Storage pools will also need to acquire parameters + to define the pools to be used for copies. + + Notes: I would commit some of my developers' time if we can agree + on the design and behavior. + +Item 9: Implement new {Client}Run{Before|After}Job feature. + Date: 26 September 2005 + Origin: Phil Stracchino + Status: Done. This has been implemented by Eric Bollengier + + What: Some time ago, there was a discussion of RunAfterJob and + ClientRunAfterJob, and the fact that they do not run after failed + jobs. At the time, there was a suggestion to add a + RunAfterFailedJob directive (and, presumably, a matching + ClientRunAfterFailedJob directive), but to my knowledge these + were never implemented. + + The current implementation doesn't permit to add new feature easily. + + An alternate way of approaching the problem has just occurred to + me. Suppose the RunBeforeJob and RunAfterJob directives were + expanded in a manner like this example: + + RunScript { + Command = "/opt/bacula/etc/checkhost %c" + RunsOnClient = No # default + AbortJobOnError = Yes # default + RunsWhen = Before + } + RunScript { + Command = c:/bacula/systemstate.bat + RunsOnClient = yes + AbortJobOnError = No + RunsWhen = After + RunsOnFailure = yes + } + + RunScript { + Command = c:/bacula/deletestatefile.bat + Target = rico-fd + RunsWhen = Always + } + + It's now possible to specify more than 1 command per Job. + (you can stop your database and your webserver without a script) + + ex : + Job { + Name = "Client1" + JobDefs = "DefaultJob" + Write Bootstrap = "/tmp/bacula/var/bacula/working/Client1.bsr" + FileSet = "Minimal" + + RunBeforeJob = "echo test before ; echo test before2" + RunBeforeJob = "echo test before (2nd time)" + RunBeforeJob = "echo test before (3rd time)" + RunAfterJob = "echo test after" + ClientRunAfterJob = "echo test after client" + + RunScript { + Command = "echo test RunScript in error" + Runsonclient = yes + RunsOnSuccess = no + RunsOnFailure = yes + RunsWhen = After # never by default + } + RunScript { + Command = "echo test RunScript on success" + Runsonclient = yes + RunsOnSuccess = yes # default + RunsOnFailure = no # default + RunsWhen = After + } + } + + Why: It would be a significant change to the structure of the + directives, but allows for a lot more flexibility, including + RunAfter commands that will run regardless of whether the job + succeeds, or RunBefore tasks that still allow the job to run even + if that specific RunBefore fails. + + Notes: (More notes from Phil, Kern, David and Eric) + I would prefer to have a single new Resource called + RunScript. + + RunsWhen = After|Before|Always + RunsAtJobLevels = All|Full|Diff|Inc # not yet implemented + + The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives + could be optional, and possibly RunWhen as well. + + AbortJobOnError would be ignored unless RunsWhen was set to Before + and would default to Yes if omitted. + If AbortJobOnError was set to No, failure of the script + would still generate a warning. + + RunsOnSuccess would be ignored unless RunsWhen was set to After + (or RunsBeforeJob set to No), and default to Yes. + + RunsOnFailure would be ignored unless RunsWhen was set to After, + and default to No. + + Allow having the before/after status on the script command + line so that the same script can be used both before/after. + +Item 10: Merge multiple backups (Synthetic Backup or Consolidation). + Origin: Marc Cousin and Eric Bollengier + Date: 15 November 2005 + Status: Waiting implementation. Depends on first implementing + project Item 2 (Migration). + + What: A merged backup is a backup made without connecting to the Client. + It would be a Merge of existing backups into a single backup. + In effect, it is like a restore but to the backup medium. + + For instance, say that last Sunday we made a full backup. Then + all week long, we created incremental backups, in order to do + them fast. Now comes Sunday again, and we need another full. + The merged backup makes it possible to do instead an incremental + backup (during the night for instance), and then create a merged + backup during the day, by using the full and incrementals from + the week. The merged backup will be exactly like a full made + Sunday night on the tape, but the production interruption on the + Client will be minimal, as the Client will only have to send + incrementals. + + In fact, if it's done correctly, you could merge all the + Incrementals into single Incremental, or all the Incrementals + and the last Differential into a new Differential, or the Full, + last differential and all the Incrementals into a new Full + backup. And there is no need to involve the Client. + + Why: The benefit is that : + - the Client just does an incremental ; + - the merged backup on tape is just as a single full backup, + and can be restored very fast. + + This is also a way of reducing the backup data since the old + data can then be pruned (or not) from the catalog, possibly + allowing older volumes to be recycled + +Item 11: Deletion of Disk-Based Bacula Volumes + Date: Nov 25, 2005 + Origin: Ross Boylan (edited + by Kern) + Status: + + What: Provide a way for Bacula to automatically remove Volumes + from the filesystem, or optionally to truncate them. + Obviously, the Volume must be pruned prior removal. + + Why: This would allow users more control over their Volumes and + prevent disk based volumes from consuming too much space. + + Notes: The following two directives might do the trick: + + Volume Data Retention =