+Item 6: Allow FD to initiate a backup
+ Origin: Frank Volf (frank at deze dot org)
+ Date: 17 November 2005
+ Status:
+
+ What: Provide some means, possibly by a restricted console that
+ allows a FD to initiate a backup, and that uses the connection
+ established by the FD to the Director for the backup so that
+ a Director that is firewalled can do the backup.
+
+ Why: Makes backup of laptops much easier.
+
+Item 7: Improve Bacula's tape and drive usage and cleaning management.
+ Date: 8 November 2005, November 11, 2005
+ Origin: Adam Thornton <athornton at sinenomine dot net>,
+ Arno Lehmann <al at its-lehmann dot de>
+ Status:
+
+ What: Make Bacula manage tape life cycle information, tape reuse
+ times and drive cleaning cycles.
+
+ Why: All three parts of this project are important when operating
+ backups.
+ We need to know which tapes need replacement, and we need to
+ make sure the drives are cleaned when necessary. While many
+ tape libraries and even autoloaders can handle all this
+ automatically, support by Bacula can be helpful for smaller
+ (older) libraries and single drives. Limiting the number of
+ times a tape is used might prevent tape errors when using
+ tapes until the drives can't read it any more. Also, checking
+ drive status during operation can prevent some failures (as I
+ [Arno] had to learn the hard way...)
+
+ Notes: First, Bacula could (and even does, to some limited extent)
+ record tape and drive usage. For tapes, the number of mounts,
+ the amount of data, and the time the tape has actually been
+ running could be recorded. Data fields for Read and Write
+ time and Number of mounts already exist in the catalog (I'm
+ not sure if VolBytes is the sum of all bytes ever written to
+ that volume by Bacula). This information can be important
+ when determining which media to replace. The ability to mark
+ Volumes as "used up" after a given number of write cycles
+ should also be implemented so that a tape is never actually
+ worn out. For the tape drives known to Bacula, similar
+ information is interesting to determine the device status and
+ expected life time: Time it's been Reading and Writing, number
+ of tape Loads / Unloads / Errors. This information is not yet
+ recorded as far as I [Arno] know. A new volume status would
+ be necessary for the new state, like "Used up" or "Worn out".
+ Volumes with this state could be used for restores, but not
+ for writing. These volumes should be migrated first (assuming
+ migration is implemented) and, once they are no longer needed,
+ could be moved to a Trash pool.
+
+ The next step would be to implement a drive cleaning setup.
+ Bacula already has knowledge about cleaning tapes. Once it
+ has some information about cleaning cycles (measured in drive
+ run time, number of tapes used, or calender days, for example)
+ it can automatically execute tape cleaning (with an
+ autochanger, obviously) or ask for operator assistance loading
+ a cleaning tape.
+
+ The final step would be to implement TAPEALERT checks not only
+ when changing tapes and only sending the information to the
+ administrator, but rather checking after each tape error,
+ checking on a regular basis (for example after each tape
+ file), and also before unloading and after loading a new tape.
+ Then, depending on the drives TAPEALERT state and the known
+ drive cleaning state Bacula could automatically schedule later
+ cleaning, clean immediately, or inform the operator.
+
+ Implementing this would perhaps require another catalog change
+ and perhaps major changes in SD code and the DIR-SD protocol,
+ so I'd only consider this worth implementing if it would
+ actually be used or even needed by many people.
+
+ Implementation of these projects could happen in three distinct
+ sub-projects: Measuring Tape and Drive usage, retiring
+ volumes, and handling drive cleaning and TAPEALERTs.
+
+Item 8: Implement creation and maintenance of copy pools
+ Date: 27 November 2005
+ Origin: David Boyes (dboyes at sinenomine dot net)
+ Status:
+
+ What: I would like Bacula to have the capability to write copies
+ of backed-up data on multiple physical volumes selected
+ from different pools without transferring the data
+ multiple times, and to accept any of the copy volumes
+ as valid for restore.
+
+ Why: In many cases, businesses are required to keep offsite
+ copies of backup volumes, or just wish for simple
+ protection against a human operator dropping a storage
+ volume and damaging it. The ability to generate multiple
+ volumes in the course of a single backup job allows
+ customers to simple check out one copy and send it
+ offsite, marking it as out of changer or otherwise
+ unavailable. Currently, the library and magazine
+ management capability in Bacula does not make this process
+ simple.
+
+ Restores would use the copy of the data on the first
+ available volume, in order of copy pool chain definition.
+
+ This is also a major scalability issue -- as the number of
+ clients increases beyond several thousand, and the volume
+ of data increases, transferring the data multiple times to
+ produce additional copies of the backups will become
+ physically impossible due to transfer speed
+ issues. Generating multiple copies at server side will
+ become the only practical option.
+
+ How: I suspect that this will require adding a multiplexing
+ SD that appears to be a SD to a specific FD, but 1-n FDs
+ to the specific back end SDs managing the primary and copy
+ pools. Storage pools will also need to acquire parameters
+ to define the pools to be used for copies.
+
+ Notes: I would commit some of my developers' time if we can agree
+ on the design and behavior.
+
+Item 9: Implement new {Client}Run{Before|After}Job feature.
+ Date: 26 September 2005
+ Origin: Phil Stracchino
+ Status: Done. This has been implemented by Eric Bollengier
+
+ What: Some time ago, there was a discussion of RunAfterJob and
+ ClientRunAfterJob, and the fact that they do not run after failed
+ jobs. At the time, there was a suggestion to add a
+ RunAfterFailedJob directive (and, presumably, a matching
+ ClientRunAfterFailedJob directive), but to my knowledge these
+ were never implemented.
+
+ The current implementation doesn't permit to add new feature easily.
+
+ An alternate way of approaching the problem has just occurred to
+ me. Suppose the RunBeforeJob and RunAfterJob directives were
+ expanded in a manner like this example:
+
+ RunScript {
+ Command = "/opt/bacula/etc/checkhost %c"
+ RunsOnClient = No # default
+ AbortJobOnError = Yes # default
+ RunsWhen = Before
+ }
+ RunScript {
+ Command = c:/bacula/systemstate.bat
+ RunsOnClient = yes
+ AbortJobOnError = No
+ RunsWhen = After
+ RunsOnFailure = yes
+ }
+
+ RunScript {
+ Command = c:/bacula/deletestatefile.bat
+ Target = rico-fd
+ RunsWhen = Always
+ }
+
+ It's now possible to specify more than 1 command per Job.
+ (you can stop your database and your webserver without a script)
+
+ ex :
+ Job {
+ Name = "Client1"
+ JobDefs = "DefaultJob"
+ Write Bootstrap = "/tmp/bacula/var/bacula/working/Client1.bsr"
+ FileSet = "Minimal"
+
+ RunBeforeJob = "echo test before ; echo test before2"
+ RunBeforeJob = "echo test before (2nd time)"
+ RunBeforeJob = "echo test before (3rd time)"
+ RunAfterJob = "echo test after"
+ ClientRunAfterJob = "echo test after client"
+
+ RunScript {
+ Command = "echo test RunScript in error"
+ Runsonclient = yes
+ RunsOnSuccess = no
+ RunsOnFailure = yes
+ RunsWhen = After # never by default
+ }
+ RunScript {
+ Command = "echo test RunScript on success"
+ Runsonclient = yes
+ RunsOnSuccess = yes # default
+ RunsOnFailure = no # default
+ RunsWhen = After
+ }
+ }
+
+ Why: It would be a significant change to the structure of the
+ directives, but allows for a lot more flexibility, including
+ RunAfter commands that will run regardless of whether the job
+ succeeds, or RunBefore tasks that still allow the job to run even
+ if that specific RunBefore fails.
+
+ Notes: (More notes from Phil, Kern, David and Eric)
+ I would prefer to have a single new Resource called
+ RunScript.
+
+ RunsWhen = After|Before|Always
+ RunsAtJobLevels = All|Full|Diff|Inc # not yet implemented
+
+ The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives
+ could be optional, and possibly RunWhen as well.
+
+ AbortJobOnError would be ignored unless RunsWhen was set to Before
+ and would default to Yes if omitted.
+ If AbortJobOnError was set to No, failure of the script
+ would still generate a warning.
+
+ RunsOnSuccess would be ignored unless RunsWhen was set to After
+ (or RunsBeforeJob set to No), and default to Yes.
+
+ RunsOnFailure would be ignored unless RunsWhen was set to After,
+ and default to No.
+
+ Allow having the before/after status on the script command
+ line so that the same script can be used both before/after.
+
+Item 10: Merge multiple backups (Synthetic Backup or Consolidation).
+ Origin: Marc Cousin and Eric Bollengier
+ Date: 15 November 2005
+ Status: Waiting implementation. Depends on first implementing
+ project Item 2 (Migration).
+
+ What: A merged backup is a backup made without connecting to the Client.
+ It would be a Merge of existing backups into a single backup.
+ In effect, it is like a restore but to the backup medium.
+
+ For instance, say that last Sunday we made a full backup. Then
+ all week long, we created incremental backups, in order to do
+ them fast. Now comes Sunday again, and we need another full.
+ The merged backup makes it possible to do instead an incremental
+ backup (during the night for instance), and then create a merged
+ backup during the day, by using the full and incrementals from
+ the week. The merged backup will be exactly like a full made
+ Sunday night on the tape, but the production interruption on the
+ Client will be minimal, as the Client will only have to send
+ incrementals.
+
+ In fact, if it's done correctly, you could merge all the
+ Incrementals into single Incremental, or all the Incrementals
+ and the last Differential into a new Differential, or the Full,
+ last differential and all the Incrementals into a new Full
+ backup. And there is no need to involve the Client.
+
+ Why: The benefit is that :
+ - the Client just does an incremental ;
+ - the merged backup on tape is just as a single full backup,
+ and can be restored very fast.
+
+ This is also a way of reducing the backup data since the old
+ data can then be pruned (or not) from the catalog, possibly
+ allowing older volumes to be recycled
+
+Item 11: Deletion of Disk-Based Bacula Volumes
+ Date: Nov 25, 2005
+ Origin: Ross Boylan <RossBoylan at stanfordalumni dot org> (edited
+ by Kern)
+ Status:
+
+ What: Provide a way for Bacula to automatically remove Volumes
+ from the filesystem, or optionally to truncate them.
+ Obviously, the Volume must be pruned prior removal.
+
+ Why: This would allow users more control over their Volumes and
+ prevent disk based volumes from consuming too much space.
+
+ Notes: The following two directives might do the trick:
+
+ Volume Data Retention = <time period>
+ Remove Volume After = <time period>
+
+ The migration project should also remove a Volume that is
+ migrated. This might also work for tape Volumes.
+
+Item 12: Directive/mode to backup only file changes, not entire file
+ Date: 11 November 2005
+ Origin: Joshua Kugler <joshua dot kugler at uaf dot edu>
+ Marek Bajon <mbajon at bimsplus dot com dot pl>
+ Status:
+
+ What: Currently when a file changes, the entire file will be backed up in
+ the next incremental or full backup. To save space on the tapes
+ it would be nice to have a mode whereby only the changes to the
+ file would be backed up when it is changed.
+
+ Why: This would save lots of space when backing up large files such as
+ logs, mbox files, Outlook PST files and the like.
+
+ Notes: This would require the usage of disk-based volumes as comparing
+ files would not be feasible using a tape drive.
+
+Item 13: Multiple threads in file daemon for the same job
+ Date: 27 November 2005
+ Origin: Ove Risberg (Ove.Risberg at octocode dot com)
+ Status:
+
+ What: I want the file daemon to start multiple threads for a backup
+ job so the fastest possible backup can be made.
+
+ The file daemon could parse the FileSet information and start
+ one thread for each File entry located on a separate
+ filesystem.
+
+ A configuration option in the job section should be used to
+ enable or disable this feature. The configuration option could
+ specify the maximum number of threads in the file daemon.
+
+ If the theads could spool the data to separate spool files
+ the restore process will not be much slower.
+
+ Why: Multiple concurrent backups of a large fileserver with many
+ disks and controllers will be much faster.
+
+ Notes: I am willing to try to implement this but I will probably
+ need some help and advice. (No problem -- Kern)
+
+Item 14: Implement red/black binary tree routines.
+ Date: 28 October 2005
+ Origin: Kern
+ Status: Class code is complete. Code needs to be integrated into
+ restore tree code.
+
+ What: Implement a red/black binary tree class. This could
+ then replace the current binary insert/search routines
+ used in the restore in memory tree. This could significantly
+ speed up the creation of the in memory restore tree.
+
+ Why: Performance enhancement.
+
+Item 15: Add support for FileSets in user directories CACHEDIR.TAG
+ Origin: Norbert Kiesel <nkiesel at tbdnetworks dot com>
+ Date: 21 November 2005
+ Status: (I think this is better done using a Python event that I
+ will implement in version 1.39.x).
+
+ What: CACHDIR.TAG is a proposal for identifying directories which
+ should be ignored for archiving/backup. It works by ignoring
+ directory trees which have a file named CACHEDIR.TAG with a
+ specific content. See
+ http://www.brynosaurus.com/cachedir/spec.html
+ for details.
+
+ From Peter Eriksson:
+ I suggest that if this is implemented (I've also asked for this
+ feature some year ago) that it is made compatible with Legato
+ Networkers ".nsr" files where you can specify a lot of options on
+ how to handle files/directories (including denying further
+ parsing of .nsr files lower down into the directory trees). A
+ PDF version of the .nsr man page can be viewed at:
+
+ http://www.ifm.liu.se/~peter/nsr.pdf
+
+ Why: It's a nice alternative to "exclude" patterns for directories
+ which don't have regular pathnames. Also, it allows users to
+ control backup for themselves. Implementation should be pretty
+ simple. GNU tar >= 1.14 or so supports it, too.
+
+ Notes: I envision this as an optional feature to a fileset
+ specification.
+
+
+Item 16: Implement extraction of Win32 BackupWrite data.
+ Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
+ Date: 28 October 2005
+ Status: Done. Assigned to Thorsten. Implemented in current CVS
+
+ What: This provides the Bacula File daemon with code that
+ can pick apart the stream output that Microsoft writes
+ for BackupWrite data, and thus the data can be read
+ and restored on non-Win32 machines.
+
+ Why: BackupWrite data is the portable=no option in Win32
+ FileSets, and in previous Baculas, this data could
+ only be extracted using a Win32 FD. With this new code,
+ the Windows data can be extracted and restored on
+ any OS.
+
+
+Item 18: Implement a Python interface to the Bacula catalog.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
+ What: Implement an interface for Python scripts to access
+ the catalog through Bacula.
+
+ Why: This will permit users to customize Bacula through
+ Python scripts.
+
+Item 18: Archival (removal) of User Files to Tape
+
+ Date: Nov. 24/2005
+
+ Origin: Ray Pengelly [ray at biomed dot queensu dot ca
+ Status:
+
+ What: The ability to archive data to storage based on certain parameters
+ such as age, size, or location. Once the data has been written to
+ storage and logged it is then pruned from the originating
+ filesystem. Note! We are talking about user's files and not
+ Bacula Volumes.
+
+ Why: This would allow fully automatic storage management which becomes
+ useful for large datastores. It would also allow for auto-staging
+ from one media type to another.
+
+ Example 1) Medical imaging needs to store large amounts of data.
+ They decide to keep data on their servers for 6 months and then put
+ it away for long term storage. The server then finds all files
+ older than 6 months writes them to tape. The files are then removed
+ from the server.
+
+ Example 2) All data that hasn't been accessed in 2 months could be
+ moved from high-cost, fibre-channel disk storage to a low-cost
+ large-capacity SATA disk storage pool which doesn't have as quick of
+ access time. Then after another 6 months (or possibly as one
+ storage pool gets full) data is migrated to Tape.
+
+Item 19: Add Plug-ins to the FileSet Include statements.
+ Date: 28 October 2005
+ Origin: