Projects: Bacula Projects Roadmap 28 October 2005 Below, you will find more information on future projects: Item 1: Implement a Migration job type that will move the job data from one device to another. Origin: Sponsored by Riege Sofware International GmbH. Contact: Daniel Holtkamp Status: Partially coded in 1.37 -- much more to do. Assigned to Kern. What: The ability to copy, move, or archive data that is on a device to another device is very important. Why: An ISP might want to backup to disk, but after 30 days migrate the data to tape backup and delete it from disk. Bacula should be able to handle this automatically. It needs to know what was put where, and when, and what to migrate -- it is a bit like retention periods. Doing so would allow space to be freed up for current backups while maintaining older data on tape drives. Notes: Migration could be triggered by: Number of Jobs Number of Volumes Age of Jobs Highwater size (keep total size) Lowwater mark Item 2: Implement a Bacula GUI/management tool using Python and Qt. Origin: Kern Status: What: Implement a Bacula console, and management tools using Python and Qt. Why: Don't we already have a wxWidgets GUI? Yes, but it is written in C++ and changes to the user interface must be hand tailored using C++ code. By developing the user interface using Qt designer, the interface can be very easily updated and most of the new Python code will be automatically created. The user interface changes become very simple, and only the new features must be implement. In addition, the code will be in Python, which will give many more users easy (or easier) access to making additions or modifications. Item 3: Implement a Python interface to the Bacula catalog. Origin: Kern Status: What: Implement an interface for Python scripts to access the catalog through Bacula. Why: This will permit users to customize Bacula through Python scripts. Item 4: Implement more Python events in Bacula. Origin: Status: What: Allow Python scripts to be called at more places within Bacula and provide additional access to Bacula internal variables. Why: This will permit users to customize Bacula through Python scripts. Notes: Recycle event Scratch pool event NeedVolume event Item 5: Implement Base jobs. Origin: Kern Status: What: A base job is sort of like a Full save except that you will want the FileSet to contain only files that are unlikely to change in the future (i.e. a snapshot of most of your system after installing it). After the base job has been run, when you are doing a Full save, you specify one or more Base jobs to be used. All files that have been backed up in the Base job/jobs but not modified will then be excluded from the backup. During a restore, the Base jobs will be automatically pulled in where necessary. Why: This is something none of the competition does, as far as we know (except perhpas BackupPC, which is a Perl program that saves to disk only). It is big win for the user, it makes Bacula stand out as offering a unique optimization that immediately saves time and money. Basically, imagine that you have 100 nearly identical Windows or Linux machine containing the OS and user files. Now for the OS part, a Base job will be backed up once, and rather than making 100 copies of the OS, there will be only one. If one or more of the systems have some files updated, no problem, they will be automatically restored. Notes: Huge savings in tape usage even for a single machine. Will require more resources because the DIR must send FD a list of files/attribs, and the FD must search the list and compare it for each file to be saved. Item 6: Add Plug-ins to the FileSet Include statements. Origin: Status: Partially coded in 1.37 -- much more to do. What: Allow users to specify wild-card and/or regular expressions to be matched in both the Include and Exclude directives in a FileSet. At the same time, allow users to define plug-ins to be called (based on regular expression/wild-card matching). Why: This would give the users the ultimate ability to control how files are backed up/restored. A user could write a plug-in knows how to backup his Oracle database without stopping/starting it, for example. Item 7: Implement huge exclude list support using hashing. Origin: Kern Status: What: Allow users to specify very large exclude list (currently more than about 1000 files is too many). Why: This would give the users the ability to exclude all files that are loaded with the OS (e.g. using rpms or debs). If the user can restore the base OS from CDs, there is no need to backup all those files. A complete restore would be to restore the base OS, then do a Bacula restore. By excluding the base OS files, the backup set will be *much* smaller. Item 8: Implement data encryption (as opposed to communications encryption) Origin: Sponsored by Landon and 13 contributors to EFF. Status: Landon Fuller is currently implementing this. What: Currently the data that is stored on the Volume is not encrypted. For confidentiality, encryption of data at the File daemon level is essential. Data encryption encrypts the data in the File daemon and decrypts the data in the File daemon during a restore. Why: Large sites require this. Item 9: Permit multiple Media Types in an Autochanger Origin: Status: What: Modify the Storage daemon so that multiple Media Types can be specified in an autochanger. This would be somewhat of a simplistic implementation in that each drive would still be allowed to have only one Media Type. However, the Storage daemon will ensure that only a drive with the Media Type that matches what the Director specifies is chosen. Why: This will permit user with several different drive types to make full use of their autochangers. Item 10: Allow two different autochanger definitions that refer to the same autochanger. Origin: Kern Status: What: Currently, the autochanger script is locked based on the autochanger. That is, if multiple drives are being simultaneously used, the Storage daemon ensures that only one drive at a time can access the mtx-changer script. This change would base the locking on the control device, rather than the autochanger. It would then permit two autochanger definitions for the same autochanger, but with different drives. Logically, the autochanger could then be "partitioned" for different jobs, clients, or class of jobs, and if the locking is based on the control device (e.g. /dev/sg0) the mtx-changer script will be locked appropriately. Why: This will permit users to partition autochangers for specific use. It would also permit implementation of multiple Media Types with no changes to the Storage daemon. Item 11: Implement red/black binary tree routines. Origin: Kern Status: What: Implement a red/black binary tree class. This could then replace the current binary insert/search routines used in the restore in memory tree. This could significantly speed up the creation of the in memory restore tree. Why: Performance enhancement. ============= Empty RFC form =========== Item n: One line summary ... Origin: Name and email of originator. Status: What: More detailed explanation ... Why: Why it is important ... Notes: Additional notes or features ... ============== End RFC form ============== Items completed for release 1.38.0: #4 Embedded Python Scripting (implemented in all Daemons) #5 Events that call a Python program (Implemented in all daemons, but more cleanup work to be done). #6 Select one from among Multiple Storage Devices for Job. This is already implemented in 1.37. #7 Single Job Writing to Multiple Storage Devices. This is currently implemented with a Clone feature. #- Full multiple drive Autochanger support (done in 1.37) #- Built in support for communications encryption (TLS) done by Landon Fuller. # Support for Unicode characters (via UTF-8) on Win32 machines thanks to Thorsten Engel. Item 8: Break the one-to-one Relationship between a Job and a Specific Storage Device (or Devices if #10 is implemented). Completed items from last year's list: Item 1: Multiple simultaneous Jobs. (done) Item 3: Write the bscan program -- also write a bcopy program (done). Item 5: Implement Label templates (done). Item 6: Write a regression script (done) Item 9: Add SSL to daemon communications (done by Landon Fuller) Item 10: Define definitive tape format (done) Item 3: GUI for interactive restore. Partially Implemented in 1.34 Note, there is now a complete Webmin plugin, a partial GNOME console, and an excellent wx-console GUI. Item 4: GUI for interactive backup Item 2: Job Data Spooling. Done: Regular expression matching. Item 10: New daemon communication protocol (this has been dropped).