X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fprojects;h=76c0e48a7647daf1df43c8cf21cc345601464427;hb=ad14d3946b4817451e48f990f8592d5eac9dca76;hp=6c3450ed3759c59c74825b8a5e4caf184c5b835c;hpb=d788c7298ec0f7da8dc629de2338f0d72a8ad838;p=bacula%2Fbacula diff --git a/bacula/projects b/bacula/projects index 6c3450ed37..76c0e48a76 100644 --- a/bacula/projects +++ b/bacula/projects @@ -1,43 +1,57 @@ Projects: Bacula Projects Roadmap - 29 November 2005 + Prioritized by user vote 07 December 2005 + Status updated 30 July 2006 Summary: -Item 1: Implement Migration that moves Jobs from one Pool to another. -Item 2: Implement extraction of Win32 BackupWrite data. -Item 3: Implement a Bacula GUI/management tool using Python. -Item 4: Implement a Python interface to the Bacula catalog. -Item 5: Implement more Python events in Bacula. -Item 6: Implement Base jobs. -Item 7: Add Plug-ins to the FileSet Include statements. -Item 8: Implement huge exclude list support using hashing. -Item 9: Implement data encryption (as opposed to comm encryption) -Item 10: Permit multiple Media Types in an Autochanger -Item 11: Allow different autochanger definitions for one autochanger. -Item 12: Implement red/black binary tree routines. -Item 13: Improve Baculas tape and drive usage and cleaning management. -Item 14: Merge multiple backups (Synthetic Backup or Consolidation). -Item 15: Automatic disabling of devices -Item 16: Directive/mode to backup only file changes, not entire file -Item 17: Quick release of FD-SD connection after backup. -Item 18: Add support for FileSets in user directories CACHEDIR.TAG -Item 19: Implement new {Client}Run{Before|After}Job feature. -Item 20: Allow FD to initiate a backup -Item 21: Multiple threads in file daemon for the same job -Item 22: Archival (removal) of User Files to Tape -Item 23: Deletion of Disk-Based BaculaVolumes -Item 24: Accurate restoration of renamed/deleted files from -Item 25: Implement creation and maintenance of copy pools +Item 1: Implement data encryption (as opposed to comm encryption) +Item 2: Implement Migration that moves Jobs from one Pool to another. +Item 3: Accurate restoration of renamed/deleted files from +Item 4: Implement a Bacula GUI/management tool using Python. +Item 5: Implement Base jobs. +Item 6: Allow FD to initiate a backup +Item 7: Improve Bacula's tape and drive usage and cleaning management. +Item 8: Implement creation and maintenance of copy pools +Item 9: Implement new {Client}Run{Before|After}Job feature. +Item 10: Merge multiple backups (Synthetic Backup or Consolidation). +Item 11: Deletion of Disk-Based Bacula Volumes +Item 12: Directive/mode to backup only file changes, not entire file +Item 13: Multiple threads in file daemon for the same job +Item 14: Implement red/black binary tree routines. +Item 15: Add support for FileSets in user directories CACHEDIR.TAG +Item 16: Implement extraction of Win32 BackupWrite data. +Item 17: Implement a Python interface to the Bacula catalog. +Item 18: Archival (removal) of User Files to Tape +Item 19: Add Plug-ins to the FileSet Include statements. +Item 20: Implement more Python events in Bacula. +Item 21: Quick release of FD-SD connection after backup. +Item 22: Permit multiple Media Types in an Autochanger +Item 23: Allow different autochanger definitions for one autochanger. +Item 24: Automatic disabling of devices +Item 25: Implement huge exclude list support using hashing. Below, you will find more information on future projects: -Item 1: Implement Migration that moves Jobs from one Pool to another. +Item 1: Implement data encryption (as opposed to comm encryption) + Date: 28 October 2005 + Origin: Sponsored by Landon and 13 contributors to EFF. + Status: Done: Landon Fuller has implemented this in 1.39.x. + + What: Currently the data that is stored on the Volume is not + encrypted. For confidentiality, encryption of data at + the File daemon level is essential. + Data encryption encrypts the data in the File daemon and + decrypts the data in the File daemon during a restore. + + Why: Large sites require this. + +Item 2: Implement Migration that moves Jobs from one Pool to another. Origin: Sponsored by Riege Software International GmbH. Contact: Daniel Holtkamp Date: 28 October 2005 - Status: Partially coded in 1.37 -- much more to do. Assigned to + Status: 90% complete: Working in 1.39, more to do. Assigned to Kern. What: The ability to copy, move, or archive data that is on a @@ -61,28 +75,43 @@ Item 1: Implement Migration that moves Jobs from one Pool to another. Number of Jobs Number of Volumes +Item 3: Accurate restoration of renamed/deleted files from + Incremental/Differential backups + Date: 28 November 2005 + Origin: Martin Simmons (martin at lispworks dot com) + Status: -Item 2: Implement extraction of Win32 BackupWrite data. - Origin: Thorsten Engel - Date: 28 October 2005 - Status: Assigned to Thorsten. Implemented in current CVS + What: When restoring a fileset for a specified date (including "most + recent"), Bacula should give you exactly the files and directories + that existed at the time of the last backup prior to that date. - What: This provides the Bacula File daemon with code that - can pick apart the stream output that Microsoft writes - for BackupWrite data, and thus the data can be read - and restored on non-Win32 machines. + Currently this only works if the last backup was a Full backup. + When the last backup was Incremental/Differential, files and + directories that have been renamed or deleted since the last Full + backup are not currently restored correctly. Ditto for files with + extra/fewer hard links than at the time of the last Full backup. - Why: BackupWrite data is the portable=no option in Win32 - FileSets, and in previous Baculas, this data could - only be extracted using a Win32 FD. With this new code, - the Windows data can be extracted and restored on - any OS. + Why: Incremental/Differential would be much more useful if this worked. + + Notes: Item 14 (Merging of multiple backups into a single one) seems to + rely on this working, otherwise the merged backups will not be + truly equivalent to a Full backup. + + Kern: notes shortened. This can be done without the need for + inodes. It is essentially the same as the current Verify job, + but one additional database record must be written, which does + not need any database change. + Kern: see if we can correct restoration of directories if + replace=ifnewer is set. Currently, if the directory does not + exist, a "dummy" directory is created, then when all the files + are updated, the dummy directory is newer so the real values + are not updated. -Item 3: Implement a Bacula GUI/management tool using Python. +Item 4: Implement a Bacula GUI/management tool using Python. Origin: Kern Date: 28 October 2005 - Status: + Status: Lucus is working on this for Python GTK+. What: Implement a Bacula console, and management tools using Python and Qt or GTK. @@ -101,40 +130,7 @@ Item 3: Implement a Bacula GUI/management tool using Python. Notes: This is currently being implemented using Python-GTK by Lucas Di Pentima - -Item 4: Implement a Python interface to the Bacula catalog. - Date: 28 October 2005 - Origin: Kern - Status: - - What: Implement an interface for Python scripts to access - the catalog through Bacula. - - Why: This will permit users to customize Bacula through - Python scripts. - -Item 5: Implement more Python events in Bacula. - Date: 28 October 2005 - Origin: - Status: - - What: Allow Python scripts to be called at more places - within Bacula and provide additional access to Bacula - internal variables. - - Why: This will permit users to customize Bacula through - Python scripts. - - Notes: Recycle event - Scratch pool event - NeedVolume event - MediaFull event - - Also add a way to get a listing of currently running - jobs (possibly also scheduled jobs). - - -Item 6: Implement Base jobs. +Item 5: Implement Base jobs. Date: 28 October 2005 Origin: Kern Status: @@ -168,101 +164,19 @@ Item 6: Implement Base jobs. FD a list of files/attribs, and the FD must search the list and compare it for each file to be saved. -Item 7: Add Plug-ins to the FileSet Include statements. - Date: 28 October 2005 - Origin: - Status: Partially coded in 1.37 -- much more to do. - - What: Allow users to specify wild-card and/or regular - expressions to be matched in both the Include and - Exclude directives in a FileSet. At the same time, - allow users to define plug-ins to be called (based on - regular expression/wild-card matching). - - Why: This would give the users the ultimate ability to control - how files are backed up/restored. A user could write a - plug-in knows how to backup his Oracle database without - stopping/starting it, for example. - -Item 8: Implement huge exclude list support using hashing. - Date: 28 October 2005 - Origin: Kern - Status: - - What: Allow users to specify very large exclude list (currently - more than about 1000 files is too many). - - Why: This would give the users the ability to exclude all - files that are loaded with the OS (e.g. using rpms - or debs). If the user can restore the base OS from - CDs, there is no need to backup all those files. A - complete restore would be to restore the base OS, then - do a Bacula restore. By excluding the base OS files, the - backup set will be *much* smaller. - - -Item 9: Implement data encryption (as opposed to comm encryption) - Date: 28 October 2005 - Origin: Sponsored by Landon and 13 contributors to EFF. - Status: Landon Fuller is currently implementing this. - - What: Currently the data that is stored on the Volume is not - encrypted. For confidentiality, encryption of data at - the File daemon level is essential. - Data encryption encrypts the data in the File daemon and - decrypts the data in the File daemon during a restore. - - Why: Large sites require this. - -Item 10: Permit multiple Media Types in an Autochanger - Origin: Kern - Status: Now implemented - - What: Modify the Storage daemon so that multiple Media Types - can be specified in an autochanger. This would be somewhat - of a simplistic implementation in that each drive would - still be allowed to have only one Media Type. However, - the Storage daemon will ensure that only a drive with - the Media Type that matches what the Director specifies - is chosen. - - Why: This will permit user with several different drive types - to make full use of their autochangers. - -Item 11: Allow different autochanger definitions for one autochanger. - Date: 28 October 2005 - Origin: Kern - Status: - - What: Currently, the autochanger script is locked based on - the autochanger. That is, if multiple drives are being - simultaneously used, the Storage daemon ensures that only - one drive at a time can access the mtx-changer script. - This change would base the locking on the control device, - rather than the autochanger. It would then permit two autochanger - definitions for the same autochanger, but with different - drives. Logically, the autochanger could then be "partitioned" - for different jobs, clients, or class of jobs, and if the locking - is based on the control device (e.g. /dev/sg0) the mtx-changer - script will be locked appropriately. - - Why: This will permit users to partition autochangers for specific - use. It would also permit implementation of multiple Media - Types with no changes to the Storage daemon. - -Item 12: Implement red/black binary tree routines. - Date: 28 October 2005 - Origin: Kern - Status: +Item 6: Allow FD to initiate a backup + Origin: Frank Volf (frank at deze dot org) + Date: 17 November 2005 + Status: - What: Implement a red/black binary tree class. This could - then replace the current binary insert/search routines - used in the restore in memory tree. This could significantly - speed up the creation of the in memory restore tree. + What: Provide some means, possibly by a restricted console that + allows a FD to initiate a backup, and that uses the connection + established by the FD to the Director for the backup so that + a Director that is firewalled can do the backup. - Why: Performance enhancement. + Why: Makes backup of laptops much easier. -Item 13: Improve Baculas tape and drive usage and cleaning management. +Item 7: Improve Bacula's tape and drive usage and cleaning management. Date: 8 November 2005, November 11, 2005 Origin: Adam Thornton , Arno Lehmann @@ -330,11 +244,153 @@ Item 13: Improve Baculas tape and drive usage and cleaning management. sub-projects: Measuring Tape and Drive usage, retiring volumes, and handling drive cleaning and TAPEALERTs. +Item 8: Implement creation and maintenance of copy pools + Date: 27 November 2005 + Origin: David Boyes (dboyes at sinenomine dot net) + Status: + + What: I would like Bacula to have the capability to write copies + of backed-up data on multiple physical volumes selected + from different pools without transferring the data + multiple times, and to accept any of the copy volumes + as valid for restore. + + Why: In many cases, businesses are required to keep offsite + copies of backup volumes, or just wish for simple + protection against a human operator dropping a storage + volume and damaging it. The ability to generate multiple + volumes in the course of a single backup job allows + customers to simple check out one copy and send it + offsite, marking it as out of changer or otherwise + unavailable. Currently, the library and magazine + management capability in Bacula does not make this process + simple. + + Restores would use the copy of the data on the first + available volume, in order of copy pool chain definition. + + This is also a major scalability issue -- as the number of + clients increases beyond several thousand, and the volume + of data increases, transferring the data multiple times to + produce additional copies of the backups will become + physically impossible due to transfer speed + issues. Generating multiple copies at server side will + become the only practical option. + + How: I suspect that this will require adding a multiplexing + SD that appears to be a SD to a specific FD, but 1-n FDs + to the specific back end SDs managing the primary and copy + pools. Storage pools will also need to acquire parameters + to define the pools to be used for copies. + + Notes: I would commit some of my developers' time if we can agree + on the design and behavior. + +Item 9: Implement new {Client}Run{Before|After}Job feature. + Date: 26 September 2005 + Origin: Phil Stracchino + Status: Done. This has been implemented by Eric Bollengier + + What: Some time ago, there was a discussion of RunAfterJob and + ClientRunAfterJob, and the fact that they do not run after failed + jobs. At the time, there was a suggestion to add a + RunAfterFailedJob directive (and, presumably, a matching + ClientRunAfterFailedJob directive), but to my knowledge these + were never implemented. + + The current implementation doesn't permit to add new feature easily. + + An alternate way of approaching the problem has just occurred to + me. Suppose the RunBeforeJob and RunAfterJob directives were + expanded in a manner like this example: + + RunScript { + Command = "/opt/bacula/etc/checkhost %c" + RunsOnClient = No # default + AbortJobOnError = Yes # default + RunsWhen = Before + } + RunScript { + Command = c:/bacula/systemstate.bat + RunsOnClient = yes + AbortJobOnError = No + RunsWhen = After + RunsOnFailure = yes + } + + RunScript { + Command = c:/bacula/deletestatefile.bat + Target = rico-fd + RunsWhen = Always + } + + It's now possible to specify more than 1 command per Job. + (you can stop your database and your webserver without a script) + + ex : + Job { + Name = "Client1" + JobDefs = "DefaultJob" + Write Bootstrap = "/tmp/bacula/var/bacula/working/Client1.bsr" + FileSet = "Minimal" + + RunBeforeJob = "echo test before ; echo test before2" + RunBeforeJob = "echo test before (2nd time)" + RunBeforeJob = "echo test before (3rd time)" + RunAfterJob = "echo test after" + ClientRunAfterJob = "echo test after client" + + RunScript { + Command = "echo test RunScript in error" + Runsonclient = yes + RunsOnSuccess = no + RunsOnFailure = yes + RunsWhen = After # never by default + } + RunScript { + Command = "echo test RunScript on success" + Runsonclient = yes + RunsOnSuccess = yes # default + RunsOnFailure = no # default + RunsWhen = After + } + } + + Why: It would be a significant change to the structure of the + directives, but allows for a lot more flexibility, including + RunAfter commands that will run regardless of whether the job + succeeds, or RunBefore tasks that still allow the job to run even + if that specific RunBefore fails. + + Notes: (More notes from Phil, Kern, David and Eric) + I would prefer to have a single new Resource called + RunScript. + + RunsWhen = After|Before|Always + RunsAtJobLevels = All|Full|Diff|Inc # not yet implemented + + The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives + could be optional, and possibly RunWhen as well. + + AbortJobOnError would be ignored unless RunsWhen was set to Before + and would default to Yes if omitted. + If AbortJobOnError was set to No, failure of the script + would still generate a warning. + + RunsOnSuccess would be ignored unless RunsWhen was set to After + (or RunsBeforeJob set to No), and default to Yes. + + RunsOnFailure would be ignored unless RunsWhen was set to After, + and default to No. + + Allow having the before/after status on the script command + line so that the same script can be used both before/after. -Item 14: Merge multiple backups (Synthetic Backup or Consolidation). +Item 10: Merge multiple backups (Synthetic Backup or Consolidation). Origin: Marc Cousin and Eric Bollengier Date: 15 November 2005 - Status: Depends on first implementing project Item 1 (Migration). + Status: Waiting implementation. Depends on first implementing + project Item 2 (Migration). What: A merged backup is a backup made without connecting to the Client. It would be a Merge of existing backups into a single backup. @@ -366,38 +422,32 @@ Item 14: Merge multiple backups (Synthetic Backup or Consolidation). data can then be pruned (or not) from the catalog, possibly allowing older volumes to be recycled -Item 15: Automatic disabling of devices - Date: 2005-11-11 - Origin: Peter Eriksson - Status: +Item 11: Deletion of Disk-Based Bacula Volumes + Date: Nov 25, 2005 + Origin: Ross Boylan (edited + by Kern) + Status: - What: After a configurable amount of fatal errors with a tape drive - Bacula should automatically disable further use of a certain - tape drive. There should also be "disable"/"enable" commands in - the "bconsole" tool. + What: Provide a way for Bacula to automatically remove Volumes + from the filesystem, or optionally to truncate them. + Obviously, the Volume must be pruned prior removal. - Why: On a multi-drive jukebox there is a possibility of tape drives - going bad during large backups (needing a cleaning tape run, - tapes getting stuck). It would be advantageous if Bacula would - automatically disable further use of a problematic tape drive - after a configurable amount of errors has occurred. + Why: This would allow users more control over their Volumes and + prevent disk based volumes from consuming too much space. - An example: I have a multi-drive jukebox (6 drives, 380+ slots) - where tapes occasionally get stuck inside the drive. Bacula will - notice that the "mtx-changer" command will fail and then fail - any backup jobs trying to use that drive. However, it will still - keep on trying to run new jobs using that drive and fail - - forever, and thus failing lots and lots of jobs... Since we have - many drives Bacula could have just automatically disabled - further use of that drive and used one of the other ones - instead. + Notes: The following two directives might do the trick: + Volume Data Retention =