X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fprojects;h=fa2e4e824e2f6e79933b5f66e85af62ccb68d7b1;hb=333aa5a0fa52106f25652cebf5f3edee90a62776;hp=a0c71a60e9f6562ea792c1f8e529808dacfc1848;hpb=c5c69c23bd6e8bbe847e7b0f9744dedf0fef4f8e;p=bacula%2Fbacula diff --git a/bacula/projects b/bacula/projects index a0c71a60e9..fa2e4e824e 100644 --- a/bacula/projects +++ b/bacula/projects @@ -1,10 +1,108 @@ Projects: Bacula Projects Roadmap - 05 April 2004 + 08 November 2005 -Item 1: Implement Base jobs. +Below, you will find more information on future projects: +Item 1: Implement a Migration job type that will move the job + data from one device to another. + Origin: Sponsored by Riege Sofware International GmbH. Contact: + Daniel Holtkamp + Date: 28 October 2005 + Status: Partially coded in 1.37 -- much more to do. Assigned to + Kern. + + What: The ability to copy, move, or archive data that is on a + device to another device is very important. + + Why: An ISP might want to backup to disk, but after 30 days + migrate the data to tape backup and delete it from + disk. Bacula should be able to handle this + automatically. It needs to know what was put where, + and when, and what to migrate -- it is a bit like + retention periods. Doing so would allow space to be + freed up for current backups while maintaining older + data on tape drives. + + Notes: Migration could be triggered by: + Number of Jobs + Number of Volumes + Age of Jobs + Highwater size (keep total size) + Lowwater mark + +Item 2: Implement extraction of Win32 BackupWrite data. + Origin: Thorsten Engel + Date: 28 October 2005 + Status: Assigned to Thorsten. Implemented in current CVS + + What: This provides the Bacula File daemon with code that + can pick apart the stream output that Microsoft writes + for BackupWrite data, and thus the data can be read + and restored on non-Win32 machines. + + Why: BackupWrite data is the portable=no option in Win32 + FileSets, and in previous Baculas, this data could + only be extracted using a Win32 FD. With this new code, + the Windows data can be extracted and restored on + any OS. + + +Item 3: Implement a Bacula GUI/management tool using Python + and Qt. + + Origin: Kern + Date: 28 October 2005 + Status: + + What: Implement a Bacula console, and management tools + using Python and Qt. + + Why: Don't we already have a wxWidgets GUI? Yes, but + it is written in C++ and changes to the user interface + must be hand tailored using C++ code. By developing + the user interface using Qt designer, the interface + can be very easily updated and most of the new Python + code will be automatically created. The user interface + changes become very simple, and only the new features + must be implement. In addition, the code will be in + Python, which will give many more users easy (or easier) + access to making additions or modifications. + +Item 4: Implement a Python interface to the Bacula catalog. + Date: 28 October 2005 + Origin: Kern + Status: + + What: Implement an interface for Python scripts to access + the catalog through Bacula. + + Why: This will permit users to customize Bacula through + Python scripts. + +Item 5: Implement more Python events in Bacula. + Date: 28 October 2005 + Origin: + Status: + + What: Allow Python scripts to be called at more places + within Bacula and provide additional access to Bacula + internal variables. + + Why: This will permit users to customize Bacula through + Python scripts. + + Notes: Recycle event + Scratch pool event + NeedVolume event + + +Item 6: Implement Base jobs. + Date: 28 October 2005 + Origin: Kern + Status: + What: A base job is sort of like a Full save except that you will want the FileSet to contain only files that are unlikely to change in the future (i.e. a snapshot of @@ -17,7 +115,7 @@ Item 1: Implement Base jobs. pulled in where necessary. Why: This is something none of the competition does, as far as - we know (except BackupPC, which is a Perl program that + we know (except perhpas BackupPC, which is a Perl program that saves to disk only). It is big win for the user, it makes Bacula stand out as offering a unique optimization that immediately saves time and money. @@ -34,139 +132,10 @@ Item 1: Implement Base jobs. FD a list of files/attribs, and the FD must search the list and compare it for each file to be saved. - -Item 2: Job Data Spooling. -Implemented in 1.34 - - What: Make the Storage daemon use intermediate file storage to -buffer - the data to disk before writing it to the tape. - - Why: This would be a nice project and is the most requested - feature. Even though you may finish a client job - quicker by spooling to disk, you still have to - eventually get it onto tape. If intermediate disk - buffering allows us to improve write bandwidth to tape, - it may make sense. In addition, you can run multiple - simultaneous jobs all spool to disk, then the data can - be written one job at a time to the tape at full tape - speed. This keeps the tape running smoothly and - prevents blocks from different simultaneous jobs from - being intermixed on the tape, which is very inefficient - for restores. - - Notes: Need multiple spool directories. Should possibly be - able to spool by Job type, ... Possibly need high and - low spool data levels. - - -Item 3: GUI for interactive restore -Partially Implemented in 1.34 -Item 4: GUI for interactive backup - - What: The current interactive restore is implemented with a tty - interface. It would be much nicer to be able to "see" the - list of files backed up in typical GUI tree format. - The same mechanism could also be used for creating - ad-hoc backup FileSets (item 8). - - Why: Ease of use -- especially for the end user. - - Notes: Rather than implementing in Gtk, we probably should go - directly for a Browser implementation, even if doing so - meant the capability wouldn't be available until much - later. Not only is there the question of Windows - sites, most Solaris/HP/IRIX, etc, shops can't currently - run Gtk programs without installing lots of stuff - admins are very wary about. Most sysadmins will always - use the command line anyway, and the user who's doing - an interactive restore or backup of his own files will - in most cases be on a Windows machine running Exploder. - - -Item 5: Implement a Migration job type that will move the job - data from one device to another. - - What: The ability to copy, move, or archive data that is on a - device to another device is very important. - - Why: An ISP might want to backup to disk, but after 30 days - migrate the data to tape backup and delete it from - disk. Bacula should be able to handle this - automatically. It needs to know what was put where, - and when, and what to migrate -- it is a bit like - retention periods. Doing so would allow space to be - freed up for current backups while maintaining older - data on tape drives. - - Notes: Migration could be triggered by: - Number of Jobs - Number of Volumes - Age of Jobs - Highwater size (keep total size) - Lowwater mark - - -Item 6: Embedded Perl Scripting (precursor to 7). - - What: On a configuration parameter, embed the Perl language in - Bacula. - - Why: The embedded Perl scripting can be called to implement - Events such as "Volume Name needed", "End of Tape", - "Tape at x% of rated capacity", "Job started", - "Job Ended", "Job error", ... - - Notes: This needs Events. - - -Item 7: Implement Events - - What: When a particular user defined Event occurs, call the - embedded Perl interpreter. - - Why: This will provide the ultimate in user customization for - Bacula. Almost anything imaginable can be done if Events - are called at the appropriate place. - - Notes: There is a certain amount of work to be done on how - the user defines or "registers" events. - - -Item 8: Multiple Storage Devices for a Single Job - - What: Allow any Job to use more than one Storage device. - - Why: With two devices, for example, the second device could - have the next backup tape pre-mounted reducing operator - intervention in the middle of the night. - - -Item 9: Backup a Single Job Simultaneously to Multiple Storage - Devices - - What: Make two copies of the backup data at the same time. - - Why: Large shops typically do this and then take one set of - backups off-site. Some design work it needed in how to - specify the type of backup (backup, archive, ...) for - each Device. - - -Item 10: Break the one-to-one Relationship between a Job and a - Specific Storage Device (or Devices if #10 is implemented). - - What: Allow a Job to simply specify one or more MediaType, and - the Storage daemon will select a device for it. In - fact, the user should be able to specify one or more - MediaType, Storage daemon, and/or device to be used. - - Why: To allow more flexibility in large shops that have multiple - drives and/or multiple drives of different types. - - -Item 11: Add Regular Expression Matching and Plug-ins to the - FileSet Include statements. +Item 7: Add Plug-ins to the FileSet Include statements. + Date: 28 October 2005 + Origin: + Status: Partially coded in 1.37 -- much more to do. What: Allow users to specify wild-card and/or regular expressions to be matched in both the Include and @@ -179,52 +148,243 @@ Item 11: Add Regular Expression Matching and Plug-ins to the plug-in knows how to backup his Oracle database without stopping/starting it, for example. +Item 8: Implement huge exclude list support using hashing. + Date: 28 October 2005 + Origin: Kern + Status: + + What: Allow users to specify very large exclude list (currently + more than about 1000 files is too many). + + Why: This would give the users the ability to exclude all + files that are loaded with the OS (e.g. using rpms + or debs). If the user can restore the base OS from + CDs, there is no need to backup all those files. A + complete restore would be to restore the base OS, then + do a Bacula restore. By excluding the base OS files, the + backup set will be *much* smaller. -Item 12: Implement data encryption (as opposed to communications - encryption) +Item 9: Implement data encryption (as opposed to communications + encryption) + Date: 28 October 2005 + Origin: Sponsored by Landon and 13 contributors to EFF. + Status: Landon Fuller is currently implementing this. + What: Currently the data that is stored on the Volume is not encrypted. For confidentiality, encryption of data at - the File daemon level is essential. Note, communications - encryption encrypts the data when leaving the File daemon, - then decrypts the data on entry to the Storage daemon. + the File daemon level is essential. Data encryption encrypts the data in the File daemon and decrypts the data in the File daemon during a restore. Why: Large sites require this. - Notes: The only algorithm that is needed is AES. - http://csrc.nist.gov/CryptoToolkit/aes/ - - -Item 13: New daemon communication protocol. - - What: The current daemon to daemon protocol is basically an ASCII - printf() and sending the buffer. On the receiving end, the - buffer is sscanf()ed to unpack it. The new scheme would - retain the current ASCII sending, but would add an - argc, argv like table driven scanner to replace sscanf. - - Why: Named fields will permit error checking to ensure that - what is sent is what the receiver really wants. The - fields can be in any order and additional fields can be - ignored allowing better upward compatibility. Much - better checking of the types and values passed can be - done. - - Notes: These are internal improvements in the interest of the - long-term stability and evolution of the program. On - the one hand, the sooner they're done, the less code we - have to rip up when the time comes to install them. On - the other hand, they don't bring an immediately - perceptible benefit to potential users. - +Item 10: Permit multiple Media Types in an Autochanger + Origin: + Status: + + What: Modify the Storage daemon so that multiple Media Types + can be specified in an autochanger. This would be somewhat + of a simplistic implementation in that each drive would + still be allowed to have only one Media Type. However, + the Storage daemon will ensure that only a drive with + the Media Type that matches what the Director specifies + is chosen. + + Why: This will permit user with several different drive types + to make full use of their autochangers. + +Item 11: Allow two different autochanger definitions that refer + to the same autochanger. + Date: 28 October 2005 + Origin: Kern + Status: + + What: Currently, the autochanger script is locked based on + the autochanger. That is, if multiple drives are being + simultaneously used, the Storage daemon ensures that only + one drive at a time can access the mtx-changer script. + This change would base the locking on the control device, + rather than the autochanger. It would then permit two autochanger + definitions for the same autochanger, but with different + drives. Logically, the autochanger could then be "partitioned" + for different jobs, clients, or class of jobs, and if the locking + is based on the control device (e.g. /dev/sg0) the mtx-changer + script will be locked appropriately. + + Why: This will permit users to partition autochangers for specific + use. It would also permit implementation of multiple Media + Types with no changes to the Storage daemon. + +Item 12: Implement red/black binary tree routines. + Date: 28 October 2005 + Origin: Kern + Status: + + What: Implement a red/black binary tree class. This could + then replace the current binary insert/search routines + used in the restore in memory tree. This could significantly + speed up the creation of the in memory restore tree. + + Why: Performance enhancement. + +Item 13: Let Bacula log tape usage and handle drive cleaning cycles. + Date: November 11, 2005 + Origin: Arno Lehmann + Status: + + What: Make Bacula manage tape life cycle information and drive + cleaning cycles. + + Why: Both parts of this project are important when operating backups. + We need to know which tapes need replacement, and we need to + make sure the drives are cleaned when necessary. While many + tape libraries and even autoloaders can handle all this + automatically, support by Bacula can be helpful for smaller + (older) libraries and single drives. Also, checking drive + status during operation can prevent some failures (as I had to + learn the hard way...) + + Notes: First, Bacula could (and even does, to some limited extent) + record tape and drive usage. For tapes, the number of mounts, + the amount of data, and the time the tape has actually been + running could be recorded. Data fields for Read and Write time + and Nmber of mounts already exist in the catalog (I'm not sure + if VolBytes is the sum of all bytes ever written to that volume + by Bacula). This information can be important when determining + which media to replace. For the tape drives known to Bacula, + similar information is interesting to determine the device + status and expected life time: Time it's been Reading and + Writing, number of tape Loads / Unloads / Errors. This + information is not yet recorded as far as I know. + + The next step would be implementing drive cleaning setup. + Bacula already has knowledge about cleaning tapes. Once it has + some information about cleaning cycles (measured in drive run + time, number of tapes used, or calender days, for example) it + can automatically execute tape cleaning (with an autochanger, + obviously) or ask for operator assistence loading a cleaning + tape. + + The next step would be to implement TAPEALERT checks not only + when changing tapes and only sending he information to the + administrator, but rather checking after each tape error, + checking on a regular basis (for example after each tape file), + and also before unloading and after loading a new tape. Then, + depending on the drives TAPEALERT state and the know drive + cleaning state Bacula could automatically schedule later + cleaning, clean immediately, or inform the operator. + + Implementing this would perhaps require another catalog change + and perhaps major changes in SD code and the DIR-SD protocoll, + so I'd only consider this worth implementing if it would + actually be used or even needed by many people. + +Item 14: Merging of multiple backups into a single one. (Also called Synthetic + Backup or Consolidation). + + Origin: Marc Cousin and Eric Bollengier + Date: 15 November 2005 + Status: Depends on first implementing project Item 1 (Migration). + + What: A merged backup is a backup made without connecting to the Client. + It would be a Merge of existing backups into a single backup. + In effect, it is like a restore but to the backup medium. + + For instance, say that last sunday we made a full backup. Then + all week long, we created incremental backups, in order to do + them fast. Now comes sunday again, and we need another full. + The merged backup makes it possible to do instead an incremental + backup (during the night for instance), and then create a merged + backup during the day, by using the full and incrementals from + the week. The merged backup will be exactly like a full made + sunday night on the tape, but the production interruption on the + Client will be minimal, as the Client will only have to send + incrementals. + + In fact, if it's done correctly, you could merge all the + Incrementals into single Incremental, or all the Incrementals + and the last Differential into a new Differential, or the Full, + last differential and all the Incrementals into a new Full + backup. And there is no need to involve the Client. + + Why: The benefit is that : + - the Client just does an incremental ; + - the merged backup on tape is just as a single full backup, + and can be restored very fast. + + This is also a way of reducing the backup data since the old + data can then be pruned (or not) from the catalog, possibly + allowing older volumes to be recycled + +Item 1: Automatic disabling of devices + Date: 2005-11-11 + Origin: Peter Eriksson + Status: + + What: After a configurable amount of fatal errors with a tape drive + Bacula should automatically disable further use of a certain + tape drive. There should also be "disable"/"enable" commands in + the "bconsole" tool. + + Why: On a multi-drive jukebox there is a possibility of tape drives + going bad during large backups (needing a cleaning tape run, + tapes getting stuck). It would be advantageous if Bacula would + automatically disable further use of a problematic tape drive + after a configurable amount of errors has occured. + + An example: I have a multi-drive jukebox (6 drives, 380+ slots) + where tapes occasionally get stuck inside the drive. Bacula will + notice that the "mtx-changer" command will fail and then fail + any backup jobs trying to use that drive. However, it will still + keep on trying to run new jobs using that drive and fail - + forever, and thus failing lots and lots of jobs... Since we have + many drives Bacula could have just automatically disabled + further use of that drive and used one of the other ones + instead. + + +============= Empty RFC form =========== +Item n: One line summary ... + Date: Date submitted + Origin: Name and email of originator. + Status: + + What: More detailed explanation ... + + Why: Why it is important ... + + Notes: Additional notes or features (omit if not used) +============== End RFC form ============== + + +Items completed for release 1.38.0: +#4 Embedded Python Scripting (implemented in all Daemons) +#5 Events that call a Python program (Implemented in all + daemons, but more cleanup work to be done). +#6 Select one from among Multiple Storage Devices for Job. + This is already implemented in 1.37. +#7 Single Job Writing to Multiple Storage Devices. This is + currently implemented with a Clone feature. +#- Full multiple drive Autochanger support (done in 1.37) +#- Built in support for communications encryption (TLS) + done by Landon Fuller. +# Support for Unicode characters + (via UTF-8) on Win32 machines thanks to Thorsten Engel. +Item 8: Break the one-to-one Relationship between a Job and a + Specific Storage Device (or Devices if #10 is implemented). Completed items from last year's list: Item 1: Multiple simultaneous Jobs. (done) Item 3: Write the bscan program -- also write a bcopy program (done). Item 5: Implement Label templates (done). Item 6: Write a regression script (done) -Item 9: Add SSL to daemon communications (For now, implement with -stunnel) +Item 9: Add SSL to daemon communications (done by Landon Fuller) Item 10: Define definitive tape format (done) +Item 3: GUI for interactive restore. Partially Implemented in 1.34 + Note, there is now a complete Webmin plugin, a partial + GNOME console, and an excellent wx-console GUI. +Item 4: GUI for interactive backup +Item 2: Job Data Spooling. + Done: Regular expression matching. +Item 10: New daemon communication protocol (this has been dropped).