Projects:
- Bacula Projects Roadmap
- 18 February 2004
+ Bacula Projects Roadmap
+ 22 July 2005
-Completed items from last year's list:
-Item 1: Multiple simultaneous Jobs. (done)
-Item 3: Write the bscan program -- also write a bcopy program (done).
-Item 5: Implement Label templates (done).
-Item 6: Write a regression script (done)
-Item 9: Add SSL to daemon communications (For now, implement with stunnel)
-Item 10: Define definitive tape format (done)
-
-Item 1: Implement Base jobs.
+The following major projects are scheduled for 1.37:
- What: A base job is sort of like a Full save except that you
- will want the FileSet to contain only files that are unlikely
- to change in the future (i.e. a snapshot of most of your
- system after installing it). After the base job has been run,
- when you are doing a Full save, you can specify to exclude
- all files saved by the base job that have not been modified.
-
- Why: This is something none of the competition does, as far as we know
- (except BackupPC, which is a Perl program that saves to disk
- only). It is big win for the user, it makes Bacula stand out
- as offering a unique optimization that immediately saves time
- and money.
-
- Notes: Big savings in tape usage. Will require more resources because
- the DIR must send FD a list of files/attribs, and the FD must
- search the list and compare it for each file to be saved.
-
-Item 2: Make the Storage daemon use intermediate file storage to buffer data
- or Data Spooling.
-
- What: If data is coming into the SD too fast, buffer it to
- disk if the user has configured this option, so that tape
- shuttling or shoe-shine can be reduced.
-
- Why: This would be a nice project and is the most requested feature.
- Even though you may finish a client job quicker by spilling to
- disk, you still have to eventually get it onto tape. If
- intermediate disk buffering allows us to improve write
- bandwidth to tape, it may make sense. In addition, you can
- run multiple simultaneous jobs all spool to disk, then the
- data can be written one job at a time to the tape at full
- tape speed. This keeps the tape running smoothly and prevents
- blocks from different simultaneous jobs from being intermixed
- on the tape, which is very ineffficient for restores.
-
- Notes:
-
-Item 3: GUI for interactive restore
-Item 4: GUI for interactive backup
-
- What: The current interactive restore is implemented with a tty
- interface. It would be much nicer to be able to "see" the
- list of files backed up in typical GUI tree format.
- The same mechanism could also be used for creating
- ad-hoc backup FileSets (item 8).
-
- Why: Ease of use -- especially for the end user.
-
- Notes: Rather than implementing in Gtk, we probably should go directly
- for a Browser implementation, even if doing so meant the
- capability wouldn't be available until much later. Not only
- is there the question of Windows sites, most
- Solaris/HP/IRIX, etc, shops can't currently run Gtk programs
- without installing lots of stuff admins are very wary about.
- Real sysadmins will always use the command line anyway, and
- the user who's doing an interactive restore or backup of his
- own files will in most cases be on a Windows machine running
- Exploder.
-
-Item 5: Implement data encryption (as opposed to communications
- encryption)
-
- What: Currently the data that is stored on the Volume is not
- encrypted. For confidentiality, encryption of data at
- the File daemon level is essential. Note, communications
- encryption encrypts the data when leaving the File daemon,
- then decrypts the data on entry to the Storage daemon.
- Data encryption encrypts the data in the File daemon and
- decrypts the data in the File daemon during a restore.
-
- Why: Large sites require this.
- Notes: The only algorithm that is needed is AES.
- http://csrc.nist.gov/CryptoToolkit/aes/
+Below, you will find more information on those projects as
+well of other projects planned at a future time.
-Item 6: Implement a Migration job type that will move the job
+Item 1: Implement Base jobs.
+ Status: Voted by users not to be implemented in 1.37
+
+ What: A base job is sort of like a Full save except that you
+ will want the FileSet to contain only files that are
+ unlikely to change in the future (i.e. a snapshot of
+ most of your system after installing it). After the
+ base job has been run, when you are doing a Full save,
+ you specify one or more Base jobs to be used. All
+ files that have been backed up in the Base job/jobs but
+ not modified will then be excluded from the backup.
+ During a restore, the Base jobs will be automatically
+ pulled in where necessary.
+
+ Why: This is something none of the competition does, as far as
+ we know (except perhpas BackupPC, which is a Perl program that
+ saves to disk only). It is big win for the user, it
+ makes Bacula stand out as offering a unique
+ optimization that immediately saves time and money.
+ Basically, imagine that you have 100 nearly identical
+ Windows or Linux machine containing the OS and user
+ files. Now for the OS part, a Base job will be backed
+ up once, and rather than making 100 copies of the OS,
+ there will be only one. If one or more of the systems
+ have some files updated, no problem, they will be
+ automatically restored.
+
+ Notes: Huge savings in tape usage even for a single machine.
+ Will require more resources because the DIR must send
+ FD a list of files/attribs, and the FD must search the
+ list and compare it for each file to be saved.
+
+Item 2: Add Plug-ins to the FileSet Include statements.
+ Status: In progress in 1.37 using Python scripting.
+
+ What: Allow users to specify wild-card and/or regular
+ expressions to be matched in both the Include and
+ Exclude directives in a FileSet. At the same time,
+ allow users to define plug-ins to be called (based on
+ regular expression/wild-card matching).
+
+ Why: This would give the users the ultimate ability to control
+ how files are backed up/restored. A user could write a
+ plug-in knows how to backup his Oracle database without
+ stopping/starting it, for example.
+
+Item 3: Implement a Migration job type that will move the job
data from one device to another.
+ Status: Partially coded in 1.37 -- much more to do.
What: The ability to copy, move, or archive data that is on a
device to another device is very important.
- Why: An ISP might want to backup to disk, but after 30 days
- migrate the data to tape backup and delete it from disk.
- Bacula should be able to handle this automatically. It needs to
- know what was put where, and when, and what to migrate -- it
- is a bit like retention periods. Doing so would allow space to
- be freed up for current backups while maintaining older data on
- tape drives.
+ Why: An ISP might want to backup to disk, but after 30 days
+ migrate the data to tape backup and delete it from
+ disk. Bacula should be able to handle this
+ automatically. It needs to know what was put where,
+ and when, and what to migrate -- it is a bit like
+ retention periods. Doing so would allow space to be
+ freed up for current backups while maintaining older
+ data on tape drives.
Notes: Migration could be triggered by:
Number of Jobs
Highwater size (keep total size)
Lowwater mark
+Item 9: Implement data encryption (as opposed to communications
+ encryption)
+ Status: Landon Fuller has agreed to work on this.
+
+ What: Currently the data that is stored on the Volume is not
+ encrypted. For confidentiality, encryption of data at
+ the File daemon level is essential.
+ Data encryption encrypts the data in the File daemon and
+ decrypts the data in the File daemon during a restore.
+
+ Why: Large sites require this.
+
+
+Items completed for release 1.38.0:
+#4 Embedded Python Scripting (implemented in all Daemons)
+#5 Events that call a Python program (Implemented in all
+ daemons, but more cleanup work to be done).
+#6 Select one from among Multiple Storage Devices for Job.
+ This is already implemented in 1.37.
+#7 Single Job Writing to Multiple Storage Devices. This is
+ currently implemented with a Clone feature.
+#- Full multiple drive Autochanger support (mostly implemented).
+#- We will have built in support for communications
+ encryption (TLS) done by Landon Fuller.
+# We will most likely have support for Unicode characters
+ (via UTF-8) on Win32 machines thanks to Thorsten Engle.
+Item 8: Break the one-to-one Relationship between a Job and a
+ Specific Storage Device (or Devices if #10 is implemented).
-Item 7: New daemon communication protocol.
-
- What: The current daemon to daemon protocol is basically an ASCII
- printf() and sending the buffer. On the receiving end, the
- buffer is sscanf()ed to unpack it. The new scheme would
- retain the current ASCII sending, but would add an
- argc, argv like table driven scanner to replace sscanf.
-
- Why: Named fields will permit error checking to ensure that what is
- sent is what the receiver really wants. The fields can be in
- any order and additional fields can be ignored allowing better
- upward compatibility. Much better checking of the types and
- values passed can be done.
-
- Notes: These are internal improvements in the interest of the
- long-term stability and evolution of the program. On the one
- hand, the sooner they're done, the less code we have to rip
- up when the time comes to install them. On the other hand, they
- don't bring an immediately perceptible benefit to potential
- users.
-
-To be documented:
-Embedded Perl Scripting
-Implement events
-Multiple Storage devices for a single job
-Write to more than one device simultaneously
-Break the one-to-one relation between Storage and device
+Completed items from last year's list:
+Item 1: Multiple simultaneous Jobs. (done)
+Item 3: Write the bscan program -- also write a bcopy program (done).
+Item 5: Implement Label templates (done).
+Item 6: Write a regression script (done)
+Item 9: Add SSL to daemon communications (For now, implement with
+ stunnel)
+Item 10: Define definitive tape format (done)
+Item 3: GUI for interactive restore. Partially Implemented in 1.34
+ Note, there is now a complete Webmin plugin, a partial
+ GNOME console, and an excellent wx-console GUI.
+Item 4: GUI for interactive backup
+Item 2: Job Data Spooling.
+ Done: Regular expression matching.
+Item 10: New daemon communication protocol (this has been dropped).