Projects:
Bacula Projects Roadmap
- 05 April 2004
+ 08 November 2005
-Item 1: Implement Base jobs.
+Below, you will find more information on future projects:
+Item 1: Implement a Migration job type that will move the job
+ data from one device to another.
+ Origin: Sponsored by Riege Sofware International GmbH. Contact:
+ Daniel Holtkamp <holtkamp at riege dot com>
+ Date: 28 October 2005
+ Status: Partially coded in 1.37 -- much more to do. Assigned to
+ Kern.
+
+ What: The ability to copy, move, or archive data that is on a
+ device to another device is very important.
+
+ Why: An ISP might want to backup to disk, but after 30 days
+ migrate the data to tape backup and delete it from
+ disk. Bacula should be able to handle this
+ automatically. It needs to know what was put where,
+ and when, and what to migrate -- it is a bit like
+ retention periods. Doing so would allow space to be
+ freed up for current backups while maintaining older
+ data on tape drives.
+
+ Notes: Migration could be triggered by:
+ Number of Jobs
+ Number of Volumes
+ Age of Jobs
+ Highwater size (keep total size)
+ Lowwater mark
+
+Item 2: Implement extraction of Win32 BackupWrite data.
+ Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
+ Date: 28 October 2005
+ Status: Assigned to Thorsten. Implemented in current CVS
+
+ What: This provides the Bacula File daemon with code that
+ can pick apart the stream output that Microsoft writes
+ for BackupWrite data, and thus the data can be read
+ and restored on non-Win32 machines.
+
+ Why: BackupWrite data is the portable=no option in Win32
+ FileSets, and in previous Baculas, this data could
+ only be extracted using a Win32 FD. With this new code,
+ the Windows data can be extracted and restored on
+ any OS.
+
+
+Item 3: Implement a Bacula GUI/management tool using Python
+ and Qt.
+
+ Origin: Kern
+ Date: 28 October 2005
+ Status:
+
+ What: Implement a Bacula console, and management tools
+ using Python and Qt.
+
+ Why: Don't we already have a wxWidgets GUI? Yes, but
+ it is written in C++ and changes to the user interface
+ must be hand tailored using C++ code. By developing
+ the user interface using Qt designer, the interface
+ can be very easily updated and most of the new Python
+ code will be automatically created. The user interface
+ changes become very simple, and only the new features
+ must be implement. In addition, the code will be in
+ Python, which will give many more users easy (or easier)
+ access to making additions or modifications.
+
+Item 4: Implement a Python interface to the Bacula catalog.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
+ What: Implement an interface for Python scripts to access
+ the catalog through Bacula.
+
+ Why: This will permit users to customize Bacula through
+ Python scripts.
+
+Item 5: Implement more Python events in Bacula.
+ Date: 28 October 2005
+ Origin:
+ Status:
+
+ What: Allow Python scripts to be called at more places
+ within Bacula and provide additional access to Bacula
+ internal variables.
+
+ Why: This will permit users to customize Bacula through
+ Python scripts.
+
+ Notes: Recycle event
+ Scratch pool event
+ NeedVolume event
+
+
+Item 6: Implement Base jobs.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
What: A base job is sort of like a Full save except that you
will want the FileSet to contain only files that are
unlikely to change in the future (i.e. a snapshot of
pulled in where necessary.
Why: This is something none of the competition does, as far as
- we know (except BackupPC, which is a Perl program that
+ we know (except perhpas BackupPC, which is a Perl program that
saves to disk only). It is big win for the user, it
makes Bacula stand out as offering a unique
optimization that immediately saves time and money.
FD a list of files/attribs, and the FD must search the
list and compare it for each file to be saved.
-
-Item 2: Job Data Spooling.
-Implemented in 1.34
-
- What: Make the Storage daemon use intermediate file storage to
-buffer
- the data to disk before writing it to the tape.
-
- Why: This would be a nice project and is the most requested
- feature. Even though you may finish a client job
- quicker by spooling to disk, you still have to
- eventually get it onto tape. If intermediate disk
- buffering allows us to improve write bandwidth to tape,
- it may make sense. In addition, you can run multiple
- simultaneous jobs all spool to disk, then the data can
- be written one job at a time to the tape at full tape
- speed. This keeps the tape running smoothly and
- prevents blocks from different simultaneous jobs from
- being intermixed on the tape, which is very inefficient
- for restores.
-
- Notes: Need multiple spool directories. Should possibly be
- able to spool by Job type, ... Possibly need high and
- low spool data levels.
-
-
-Item 3: GUI for interactive restore
-Partially Implemented in 1.34
-Item 4: GUI for interactive backup
-
- What: The current interactive restore is implemented with a tty
- interface. It would be much nicer to be able to "see" the
- list of files backed up in typical GUI tree format.
- The same mechanism could also be used for creating
- ad-hoc backup FileSets (item 8).
-
- Why: Ease of use -- especially for the end user.
-
- Notes: Rather than implementing in Gtk, we probably should go
- directly for a Browser implementation, even if doing so
- meant the capability wouldn't be available until much
- later. Not only is there the question of Windows
- sites, most Solaris/HP/IRIX, etc, shops can't currently
- run Gtk programs without installing lots of stuff
- admins are very wary about. Most sysadmins will always
- use the command line anyway, and the user who's doing
- an interactive restore or backup of his own files will
- in most cases be on a Windows machine running Exploder.
-
-
-Item 5: Implement a Migration job type that will move the job
- data from one device to another.
-
- What: The ability to copy, move, or archive data that is on a
- device to another device is very important.
-
- Why: An ISP might want to backup to disk, but after 30 days
- migrate the data to tape backup and delete it from
- disk. Bacula should be able to handle this
- automatically. It needs to know what was put where,
- and when, and what to migrate -- it is a bit like
- retention periods. Doing so would allow space to be
- freed up for current backups while maintaining older
- data on tape drives.
-
- Notes: Migration could be triggered by:
- Number of Jobs
- Number of Volumes
- Age of Jobs
- Highwater size (keep total size)
- Lowwater mark
-
-
-Item 6: Embedded Perl Scripting (precursor to 7).
-
- What: On a configuration parameter, embed the Perl language in
- Bacula.
-
- Why: The embedded Perl scripting can be called to implement
- Events such as "Volume Name needed", "End of Tape",
- "Tape at x% of rated capacity", "Job started",
- "Job Ended", "Job error", ...
-
- Notes: This needs Events.
-
-
-Item 7: Implement Events
-
- What: When a particular user defined Event occurs, call the
- embedded Perl interpreter.
-
- Why: This will provide the ultimate in user customization for
- Bacula. Almost anything imaginable can be done if Events
- are called at the appropriate place.
-
- Notes: There is a certain amount of work to be done on how
- the user defines or "registers" events.
-
-
-Item 8: Multiple Storage Devices for a Single Job
-
- What: Allow any Job to use more than one Storage device.
-
- Why: With two devices, for example, the second device could
- have the next backup tape pre-mounted reducing operator
- intervention in the middle of the night.
-
-
-Item 9: Backup a Single Job Simultaneously to Multiple Storage
- Devices
-
- What: Make two copies of the backup data at the same time.
-
- Why: Large shops typically do this and then take one set of
- backups off-site. Some design work it needed in how to
- specify the type of backup (backup, archive, ...) for
- each Device.
-
-
-Item 10: Break the one-to-one Relationship between a Job and a
- Specific Storage Device (or Devices if #10 is implemented).
-
- What: Allow a Job to simply specify one or more MediaType, and
- the Storage daemon will select a device for it. In
- fact, the user should be able to specify one or more
- MediaType, Storage daemon, and/or device to be used.
-
- Why: To allow more flexibility in large shops that have multiple
- drives and/or multiple drives of different types.
-
-
-Item 11: Add Regular Expression Matching and Plug-ins to the
- FileSet Include statements.
+Item 7: Add Plug-ins to the FileSet Include statements.
+ Date: 28 October 2005
+ Origin:
+ Status: Partially coded in 1.37 -- much more to do.
What: Allow users to specify wild-card and/or regular
expressions to be matched in both the Include and
plug-in knows how to backup his Oracle database without
stopping/starting it, for example.
+Item 8: Implement huge exclude list support using hashing.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
+ What: Allow users to specify very large exclude list (currently
+ more than about 1000 files is too many).
+
+ Why: This would give the users the ability to exclude all
+ files that are loaded with the OS (e.g. using rpms
+ or debs). If the user can restore the base OS from
+ CDs, there is no need to backup all those files. A
+ complete restore would be to restore the base OS, then
+ do a Bacula restore. By excluding the base OS files, the
+ backup set will be *much* smaller.
-Item 12: Implement data encryption (as opposed to communications
- encryption)
+Item 9: Implement data encryption (as opposed to communications
+ encryption)
+ Date: 28 October 2005
+ Origin: Sponsored by Landon and 13 contributors to EFF.
+ Status: Landon Fuller is currently implementing this.
+
What: Currently the data that is stored on the Volume is not
encrypted. For confidentiality, encryption of data at
- the File daemon level is essential. Note, communications
- encryption encrypts the data when leaving the File daemon,
- then decrypts the data on entry to the Storage daemon.
+ the File daemon level is essential.
Data encryption encrypts the data in the File daemon and
decrypts the data in the File daemon during a restore.
Why: Large sites require this.
- Notes: The only algorithm that is needed is AES.
- http://csrc.nist.gov/CryptoToolkit/aes/
-
-
-Item 13: New daemon communication protocol.
-
- What: The current daemon to daemon protocol is basically an ASCII
- printf() and sending the buffer. On the receiving end, the
- buffer is sscanf()ed to unpack it. The new scheme would
- retain the current ASCII sending, but would add an
- argc, argv like table driven scanner to replace sscanf.
-
- Why: Named fields will permit error checking to ensure that
- what is sent is what the receiver really wants. The
- fields can be in any order and additional fields can be
- ignored allowing better upward compatibility. Much
- better checking of the types and values passed can be
- done.
-
- Notes: These are internal improvements in the interest of the
- long-term stability and evolution of the program. On
- the one hand, the sooner they're done, the less code we
- have to rip up when the time comes to install them. On
- the other hand, they don't bring an immediately
- perceptible benefit to potential users.
-
+Item 10: Permit multiple Media Types in an Autochanger
+ Origin:
+ Status:
+
+ What: Modify the Storage daemon so that multiple Media Types
+ can be specified in an autochanger. This would be somewhat
+ of a simplistic implementation in that each drive would
+ still be allowed to have only one Media Type. However,
+ the Storage daemon will ensure that only a drive with
+ the Media Type that matches what the Director specifies
+ is chosen.
+
+ Why: This will permit user with several different drive types
+ to make full use of their autochangers.
+
+Item 11: Allow two different autochanger definitions that refer
+ to the same autochanger.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
+ What: Currently, the autochanger script is locked based on
+ the autochanger. That is, if multiple drives are being
+ simultaneously used, the Storage daemon ensures that only
+ one drive at a time can access the mtx-changer script.
+ This change would base the locking on the control device,
+ rather than the autochanger. It would then permit two autochanger
+ definitions for the same autochanger, but with different
+ drives. Logically, the autochanger could then be "partitioned"
+ for different jobs, clients, or class of jobs, and if the locking
+ is based on the control device (e.g. /dev/sg0) the mtx-changer
+ script will be locked appropriately.
+
+ Why: This will permit users to partition autochangers for specific
+ use. It would also permit implementation of multiple Media
+ Types with no changes to the Storage daemon.
+
+Item 12: Implement red/black binary tree routines.
+ Date: 28 October 2005
+ Origin: Kern
+ Status:
+
+ What: Implement a red/black binary tree class. This could
+ then replace the current binary insert/search routines
+ used in the restore in memory tree. This could significantly
+ speed up the creation of the in memory restore tree.
+
+ Why: Performance enhancement.
+
+Item 13: Let Bacula log tape usage and handle drive cleaning cycles.
+ Date: November 11, 2005
+ Origin: Arno Lehmann <al at its-lehmann dot de>
+ Status:
+
+ What: Make Bacula manage tape life cycle information and drive
+ cleaning cycles.
+
+ Why: Both parts of this project are important when operating backups.
+ We need to know which tapes need replacement, and we need to
+ make sure the drives are cleaned when necessary. While many
+ tape libraries and even autoloaders can handle all this
+ automatically, support by Bacula can be helpful for smaller
+ (older) libraries and single drives. Also, checking drive
+ status during operation can prevent some failures (as I had to
+ learn the hard way...)
+
+ Notes: First, Bacula could (and even does, to some limited extent)
+ record tape and drive usage. For tapes, the number of mounts,
+ the amount of data, and the time the tape has actually been
+ running could be recorded. Data fields for Read and Write time
+ and Nmber of mounts already exist in the catalog (I'm not sure
+ if VolBytes is the sum of all bytes ever written to that volume
+ by Bacula). This information can be important when determining
+ which media to replace. For the tape drives known to Bacula,
+ similar information is interesting to determine the device
+ status and expected life time: Time it's been Reading and
+ Writing, number of tape Loads / Unloads / Errors. This
+ information is not yet recorded as far as I know.
+
+ The next step would be implementing drive cleaning setup.
+ Bacula already has knowledge about cleaning tapes. Once it has
+ some information about cleaning cycles (measured in drive run
+ time, number of tapes used, or calender days, for example) it
+ can automatically execute tape cleaning (with an autochanger,
+ obviously) or ask for operator assistence loading a cleaning
+ tape.
+
+ The next step would be to implement TAPEALERT checks not only
+ when changing tapes and only sending he information to the
+ administrator, but rather checking after each tape error,
+ checking on a regular basis (for example after each tape file),
+ and also before unloading and after loading a new tape. Then,
+ depending on the drives TAPEALERT state and the know drive
+ cleaning state Bacula could automatically schedule later
+ cleaning, clean immediately, or inform the operator.
+
+ Implementing this would perhaps require another catalog change
+ and perhaps major changes in SD code and the DIR-SD protocoll,
+ so I'd only consider this worth implementing if it would
+ actually be used or even needed by many people.
+
+Item 14: Merging of multiple backups into a single one. (Also called Synthetic
+ Backup or Consolidation).
+
+ Origin: Marc Cousin and Eric Bollengier
+ Date: 15 November 2005
+ Status: Depends on first implementing project Item 1 (Migration).
+
+ What: A merged backup is a backup made without connecting to the Client.
+ It would be a Merge of existing backups into a single backup.
+ In effect, it is like a restore but to the backup medium.
+
+ For instance, say that last sunday we made a full backup. Then
+ all week long, we created incremental backups, in order to do
+ them fast. Now comes sunday again, and we need another full.
+ The merged backup makes it possible to do instead an incremental
+ backup (during the night for instance), and then create a merged
+ backup during the day, by using the full and incrementals from
+ the week. The merged backup will be exactly like a full made
+ sunday night on the tape, but the production interruption on the
+ Client will be minimal, as the Client will only have to send
+ incrementals.
+
+ In fact, if it's done correctly, you could merge all the
+ Incrementals into single Incremental, or all the Incrementals
+ and the last Differential into a new Differential, or the Full,
+ last differential and all the Incrementals into a new Full
+ backup. And there is no need to involve the Client.
+
+ Why: The benefit is that :
+ - the Client just does an incremental ;
+ - the merged backup on tape is just as a single full backup,
+ and can be restored very fast.
+
+ This is also a way of reducing the backup data since the old
+ data can then be pruned (or not) from the catalog, possibly
+ allowing older volumes to be recycled
+
+Item 1: Automatic disabling of devices
+ Date: 2005-11-11
+ Origin: Peter Eriksson <peter at ifm.liu dot se>
+ Status:
+
+ What: After a configurable amount of fatal errors with a tape drive
+ Bacula should automatically disable further use of a certain
+ tape drive. There should also be "disable"/"enable" commands in
+ the "bconsole" tool.
+
+ Why: On a multi-drive jukebox there is a possibility of tape drives
+ going bad during large backups (needing a cleaning tape run,
+ tapes getting stuck). It would be advantageous if Bacula would
+ automatically disable further use of a problematic tape drive
+ after a configurable amount of errors has occured.
+
+ An example: I have a multi-drive jukebox (6 drives, 380+ slots)
+ where tapes occasionally get stuck inside the drive. Bacula will
+ notice that the "mtx-changer" command will fail and then fail
+ any backup jobs trying to use that drive. However, it will still
+ keep on trying to run new jobs using that drive and fail -
+ forever, and thus failing lots and lots of jobs... Since we have
+ many drives Bacula could have just automatically disabled
+ further use of that drive and used one of the other ones
+ instead.
+
+
+============= Empty RFC form ===========
+Item n: One line summary ...
+ Date: Date submitted
+ Origin: Name and email of originator.
+ Status:
+
+ What: More detailed explanation ...
+
+ Why: Why it is important ...
+
+ Notes: Additional notes or features (omit if not used)
+============== End RFC form ==============
+
+
+Items completed for release 1.38.0:
+#4 Embedded Python Scripting (implemented in all Daemons)
+#5 Events that call a Python program (Implemented in all
+ daemons, but more cleanup work to be done).
+#6 Select one from among Multiple Storage Devices for Job.
+ This is already implemented in 1.37.
+#7 Single Job Writing to Multiple Storage Devices. This is
+ currently implemented with a Clone feature.
+#- Full multiple drive Autochanger support (done in 1.37)
+#- Built in support for communications encryption (TLS)
+ done by Landon Fuller.
+# Support for Unicode characters
+ (via UTF-8) on Win32 machines thanks to Thorsten Engel.
+Item 8: Break the one-to-one Relationship between a Job and a
+ Specific Storage Device (or Devices if #10 is implemented).
Completed items from last year's list:
Item 1: Multiple simultaneous Jobs. (done)
Item 3: Write the bscan program -- also write a bcopy program (done).
Item 5: Implement Label templates (done).
Item 6: Write a regression script (done)
-Item 9: Add SSL to daemon communications (For now, implement with
-stunnel)
+Item 9: Add SSL to daemon communications (done by Landon Fuller)
Item 10: Define definitive tape format (done)
+Item 3: GUI for interactive restore. Partially Implemented in 1.34
+ Note, there is now a complete Webmin plugin, a partial
+ GNOME console, and an excellent wx-console GUI.
+Item 4: GUI for interactive backup
+Item 2: Job Data Spooling.
+ Done: Regular expression matching.
+Item 10: New daemon communication protocol (this has been dropped).
projects / bacula / bacula / blobdiff