3 Bacula Projects Roadmap
7 Item 1: Implement Migration that moves Jobs from one Pool to another.
8 Item 2: Implement extraction of Win32 BackupWrite data.
9 Item 3: Implement a Bacula GUI/management tool using Python.
10 Item 4: Implement a Python interface to the Bacula catalog.
11 Item 5: Implement more Python events in Bacula.
12 Item 6: Implement Base jobs.
13 Item 7: Add Plug-ins to the FileSet Include statements.
14 Item 8: Implement huge exclude list support using hashing.
15 Item 9: Implement data encryption (as opposed to comm encryption)
16 Item 10: Permit multiple Media Types in an Autochanger
17 Item 11: Allow different autochanger definitions for one autochanger.
18 Item 12: Implement red/black binary tree routines.
19 Item 13: Improve Baculas tape and drive usage and cleaning management.
20 Item 14: Merge multiple backups (Synthetic Backup or Consolidation).
21 Item 15: Automatic disabling of devices
22 Item 16: Directive/mode to backup only file changes, not entire file
23 Item 17: Quick release of FD-SD connection after backup.
24 Item 18: Add support for FileSets in user directories CACHEDIR.TAG
25 Item 19: Implement new {Client}Run{Before|After}Job feature.
26 Item 20: Allow FD to initiate a backup
27 Item 21: Multiple threads in file daemon for the same job
28 Item 22: Archival (removal) of User Files to Tape
29 Item 23: Deletion of Disk-Based BaculaVolumes
30 Item 24: Accurate restoration of renamed/deleted files from
31 Item 25: Implement creation and maintenance of copy pools
34 Below, you will find more information on future projects:
36 Item 1: Implement Migration that moves Jobs from one Pool to another.
37 Origin: Sponsored by Riege Software International GmbH. Contact:
38 Daniel Holtkamp <holtkamp at riege dot com>
40 Status: Partially coded in 1.37 -- much more to do. Assigned to
43 What: The ability to copy, move, or archive data that is on a
44 device to another device is very important.
46 Why: An ISP might want to backup to disk, but after 30 days
47 migrate the data to tape backup and delete it from
48 disk. Bacula should be able to handle this
49 automatically. It needs to know what was put where,
50 and when, and what to migrate -- it is a bit like
51 retention periods. Doing so would allow space to be
52 freed up for current backups while maintaining older
55 Notes: Riege Software have asked for the following migration
58 Highwater mark (stopped by Lowwater mark?)
60 Notes: Migration could be additionally triggered by:
65 Item 2: Implement extraction of Win32 BackupWrite data.
66 Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
68 Status: Assigned to Thorsten. Implemented in current CVS
70 What: This provides the Bacula File daemon with code that
71 can pick apart the stream output that Microsoft writes
72 for BackupWrite data, and thus the data can be read
73 and restored on non-Win32 machines.
75 Why: BackupWrite data is the portable=no option in Win32
76 FileSets, and in previous Baculas, this data could
77 only be extracted using a Win32 FD. With this new code,
78 the Windows data can be extracted and restored on
82 Item 3: Implement a Bacula GUI/management tool using Python.
87 What: Implement a Bacula console, and management tools
88 using Python and Qt or GTK.
90 Why: Don't we already have a wxWidgets GUI? Yes, but
91 it is written in C++ and changes to the user interface
92 must be hand tailored using C++ code. By developing
93 the user interface using Qt designer, the interface
94 can be very easily updated and most of the new Python
95 code will be automatically created. The user interface
96 changes become very simple, and only the new features
97 must be implement. In addition, the code will be in
98 Python, which will give many more users easy (or easier)
99 access to making additions or modifications.
101 Notes: This is currently being implemented using Python-GTK by
102 Lucas Di Pentima <lucas at lunix dot com dot ar>
105 Item 4: Implement a Python interface to the Bacula catalog.
106 Date: 28 October 2005
110 What: Implement an interface for Python scripts to access
111 the catalog through Bacula.
113 Why: This will permit users to customize Bacula through
116 Item 5: Implement more Python events in Bacula.
117 Date: 28 October 2005
121 What: Allow Python scripts to be called at more places
122 within Bacula and provide additional access to Bacula
125 Why: This will permit users to customize Bacula through
133 Also add a way to get a listing of currently running
134 jobs (possibly also scheduled jobs).
137 Item 6: Implement Base jobs.
138 Date: 28 October 2005
142 What: A base job is sort of like a Full save except that you
143 will want the FileSet to contain only files that are
144 unlikely to change in the future (i.e. a snapshot of
145 most of your system after installing it). After the
146 base job has been run, when you are doing a Full save,
147 you specify one or more Base jobs to be used. All
148 files that have been backed up in the Base job/jobs but
149 not modified will then be excluded from the backup.
150 During a restore, the Base jobs will be automatically
151 pulled in where necessary.
153 Why: This is something none of the competition does, as far as
154 we know (except perhaps BackupPC, which is a Perl program that
155 saves to disk only). It is big win for the user, it
156 makes Bacula stand out as offering a unique
157 optimization that immediately saves time and money.
158 Basically, imagine that you have 100 nearly identical
159 Windows or Linux machine containing the OS and user
160 files. Now for the OS part, a Base job will be backed
161 up once, and rather than making 100 copies of the OS,
162 there will be only one. If one or more of the systems
163 have some files updated, no problem, they will be
164 automatically restored.
166 Notes: Huge savings in tape usage even for a single machine.
167 Will require more resources because the DIR must send
168 FD a list of files/attribs, and the FD must search the
169 list and compare it for each file to be saved.
171 Item 7: Add Plug-ins to the FileSet Include statements.
172 Date: 28 October 2005
174 Status: Partially coded in 1.37 -- much more to do.
176 What: Allow users to specify wild-card and/or regular
177 expressions to be matched in both the Include and
178 Exclude directives in a FileSet. At the same time,
179 allow users to define plug-ins to be called (based on
180 regular expression/wild-card matching).
182 Why: This would give the users the ultimate ability to control
183 how files are backed up/restored. A user could write a
184 plug-in knows how to backup his Oracle database without
185 stopping/starting it, for example.
187 Item 8: Implement huge exclude list support using hashing.
188 Date: 28 October 2005
192 What: Allow users to specify very large exclude list (currently
193 more than about 1000 files is too many).
195 Why: This would give the users the ability to exclude all
196 files that are loaded with the OS (e.g. using rpms
197 or debs). If the user can restore the base OS from
198 CDs, there is no need to backup all those files. A
199 complete restore would be to restore the base OS, then
200 do a Bacula restore. By excluding the base OS files, the
201 backup set will be *much* smaller.
204 Item 9: Implement data encryption (as opposed to comm encryption)
205 Date: 28 October 2005
206 Origin: Sponsored by Landon and 13 contributors to EFF.
207 Status: Landon Fuller is currently implementing this.
209 What: Currently the data that is stored on the Volume is not
210 encrypted. For confidentiality, encryption of data at
211 the File daemon level is essential.
212 Data encryption encrypts the data in the File daemon and
213 decrypts the data in the File daemon during a restore.
215 Why: Large sites require this.
217 Item 10: Permit multiple Media Types in an Autochanger
219 Status: Now implemented
221 What: Modify the Storage daemon so that multiple Media Types
222 can be specified in an autochanger. This would be somewhat
223 of a simplistic implementation in that each drive would
224 still be allowed to have only one Media Type. However,
225 the Storage daemon will ensure that only a drive with
226 the Media Type that matches what the Director specifies
229 Why: This will permit user with several different drive types
230 to make full use of their autochangers.
232 Item 11: Allow different autochanger definitions for one autochanger.
233 Date: 28 October 2005
237 What: Currently, the autochanger script is locked based on
238 the autochanger. That is, if multiple drives are being
239 simultaneously used, the Storage daemon ensures that only
240 one drive at a time can access the mtx-changer script.
241 This change would base the locking on the control device,
242 rather than the autochanger. It would then permit two autochanger
243 definitions for the same autochanger, but with different
244 drives. Logically, the autochanger could then be "partitioned"
245 for different jobs, clients, or class of jobs, and if the locking
246 is based on the control device (e.g. /dev/sg0) the mtx-changer
247 script will be locked appropriately.
249 Why: This will permit users to partition autochangers for specific
250 use. It would also permit implementation of multiple Media
251 Types with no changes to the Storage daemon.
253 Item 12: Implement red/black binary tree routines.
254 Date: 28 October 2005
258 What: Implement a red/black binary tree class. This could
259 then replace the current binary insert/search routines
260 used in the restore in memory tree. This could significantly
261 speed up the creation of the in memory restore tree.
263 Why: Performance enhancement.
265 Item 13: Improve Baculas tape and drive usage and cleaning management.
266 Date: 8 November 2005, November 11, 2005
267 Origin: Adam Thornton <athornton at sinenomine dot net>,
268 Arno Lehmann <al at its-lehmann dot de>
271 What: Make Bacula manage tape life cycle information, tape reuse
272 times and drive cleaning cycles.
274 Why: All three parts of this project are important when operating
276 We need to know which tapes need replacement, and we need to
277 make sure the drives are cleaned when necessary. While many
278 tape libraries and even autoloaders can handle all this
279 automatically, support by Bacula can be helpful for smaller
280 (older) libraries and single drives. Limiting the number of
281 times a tape is used might prevent tape errors when using
282 tapes until the drives can't read it any more. Also, checking
283 drive status during operation can prevent some failures (as I
284 [Arno] had to learn the hard way...)
286 Notes: First, Bacula could (and even does, to some limited extent)
287 record tape and drive usage. For tapes, the number of mounts,
288 the amount of data, and the time the tape has actually been
289 running could be recorded. Data fields for Read and Write
290 time and Number of mounts already exist in the catalog (I'm
291 not sure if VolBytes is the sum of all bytes ever written to
292 that volume by Bacula). This information can be important
293 when determining which media to replace. The ability to mark
294 Volumes as "used up" after a given number of write cycles
295 should also be implemented so that a tape is never actually
296 worn out. For the tape drives known to Bacula, similar
297 information is interesting to determine the device status and
298 expected life time: Time it's been Reading and Writing, number
299 of tape Loads / Unloads / Errors. This information is not yet
300 recorded as far as I [Arno] know. A new volume status would
301 be necessary for the new state, like "Used up" or "Worn out".
302 Volumes with this state could be used for restores, but not
303 for writing. These volumes should be migrated first (assuming
304 migration is implemented) and, once they are no longer needed,
305 could be moved to a Trash pool.
307 The next step would be to implement a drive cleaning setup.
308 Bacula already has knowledge about cleaning tapes. Once it
309 has some information about cleaning cycles (measured in drive
310 run time, number of tapes used, or calender days, for example)
311 it can automatically execute tape cleaning (with an
312 autochanger, obviously) or ask for operator assistance loading
315 The final step would be to implement TAPEALERT checks not only
316 when changing tapes and only sending the information to the
317 administrator, but rather checking after each tape error,
318 checking on a regular basis (for example after each tape
319 file), and also before unloading and after loading a new tape.
320 Then, depending on the drives TAPEALERT state and the known
321 drive cleaning state Bacula could automatically schedule later
322 cleaning, clean immediately, or inform the operator.
324 Implementing this would perhaps require another catalog change
325 and perhaps major changes in SD code and the DIR-SD protocol,
326 so I'd only consider this worth implementing if it would
327 actually be used or even needed by many people.
329 Implementation of these projects could happen in three distinct
330 sub-projects: Measuring Tape and Drive usage, retiring
331 volumes, and handling drive cleaning and TAPEALERTs.
334 Item 14: Merge multiple backups (Synthetic Backup or Consolidation).
335 Origin: Marc Cousin and Eric Bollengier
336 Date: 15 November 2005
337 Status: Depends on first implementing project Item 1 (Migration).
339 What: A merged backup is a backup made without connecting to the Client.
340 It would be a Merge of existing backups into a single backup.
341 In effect, it is like a restore but to the backup medium.
343 For instance, say that last Sunday we made a full backup. Then
344 all week long, we created incremental backups, in order to do
345 them fast. Now comes Sunday again, and we need another full.
346 The merged backup makes it possible to do instead an incremental
347 backup (during the night for instance), and then create a merged
348 backup during the day, by using the full and incrementals from
349 the week. The merged backup will be exactly like a full made
350 Sunday night on the tape, but the production interruption on the
351 Client will be minimal, as the Client will only have to send
354 In fact, if it's done correctly, you could merge all the
355 Incrementals into single Incremental, or all the Incrementals
356 and the last Differential into a new Differential, or the Full,
357 last differential and all the Incrementals into a new Full
358 backup. And there is no need to involve the Client.
360 Why: The benefit is that :
361 - the Client just does an incremental ;
362 - the merged backup on tape is just as a single full backup,
363 and can be restored very fast.
365 This is also a way of reducing the backup data since the old
366 data can then be pruned (or not) from the catalog, possibly
367 allowing older volumes to be recycled
369 Item 15: Automatic disabling of devices
371 Origin: Peter Eriksson <peter at ifm.liu dot se>
374 What: After a configurable amount of fatal errors with a tape drive
375 Bacula should automatically disable further use of a certain
376 tape drive. There should also be "disable"/"enable" commands in
379 Why: On a multi-drive jukebox there is a possibility of tape drives
380 going bad during large backups (needing a cleaning tape run,
381 tapes getting stuck). It would be advantageous if Bacula would
382 automatically disable further use of a problematic tape drive
383 after a configurable amount of errors has occurred.
385 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
386 where tapes occasionally get stuck inside the drive. Bacula will
387 notice that the "mtx-changer" command will fail and then fail
388 any backup jobs trying to use that drive. However, it will still
389 keep on trying to run new jobs using that drive and fail -
390 forever, and thus failing lots and lots of jobs... Since we have
391 many drives Bacula could have just automatically disabled
392 further use of that drive and used one of the other ones
396 Item 16: Directive/mode to backup only file changes, not entire file
397 Date: 11 November 2005
398 Origin: Joshua Kugler <joshua dot kugler at uaf dot edu>
399 Marek Bajon <mbajon at bimsplus dot com dot pl>
402 What: Currently when a file changes, the entire file will be backed up in
403 the next incremental or full backup. To save space on the tapes
404 it would be nice to have a mode whereby only the changes to the
405 file would be backed up when it is changed.
407 Why: This would save lots of space when backing up large files such as
408 logs, mbox files, Outlook PST files and the like.
410 Notes: This would require the usage of disk-based volumes as comparing
411 files would not be feasible using a tape drive.
413 Item 17: Quick release of FD-SD connection after backup.
414 Origin: Frank Volf (frank at deze dot org)
415 Date: 17 November 2005
418 What: In the Bacula implementation a backup is finished after all data
419 and attributes are successfully written to storage. When using a
420 tape backup it is very annoying that a backup can take a day,
421 simply because the current tape (or whatever) is full and the
422 administrator has not put a new one in. During that time the
423 system cannot be taken off-line, because there is still an open
424 session between the storage daemon and the file daemon on the
427 Although this is a very good strategy for making "safe backups"
428 This can be annoying for e.g. laptops, that must remain
429 connected until the backup is completed.
431 Using a new feature called "migration" it will be possible to
432 spool first to harddisk (using a special 'spool' migration
433 scheme) and then migrate the backup to tape.
435 There is still the problem of getting the attributes committed.
436 If it takes a very long time to do, with the current code, the
437 job has not terminated, and the File daemon is not freed up. The
438 Storage daemon should release the File daemon as soon as all the
439 file data and all the attributes have been sent to it (the SD).
440 Currently the SD waits until everything is on tape and all the
441 attributes are transmitted to the Director before signaling
442 completion to the FD. I don't think I would have any problem
443 changing this. The reason is that even if the FD reports back to
444 the Dir that all is OK, the job will not terminate until the SD
445 has done the same thing -- so in a way keeping the SD-FD link
446 open to the very end is not really very productive ...
448 Why: Makes backup of laptops much easier.
451 Item 18: Add support for FileSets in user directories CACHEDIR.TAG
452 Origin: Norbert Kiesel <nkiesel at tbdnetworks dot com>
453 Date: 21 November 2005
456 What: CACHDIR.TAG is a proposal for identifying directories which
457 should be ignored for archiving/backup. It works by ignoring
458 directory trees which have a file named CACHEDIR.TAG with a
459 specific content. See
460 http://www.brynosaurus.com/cachedir/spec.html
464 I suggest that if this is implemented (I've also asked for this
465 feature some year ago) that it is made compatible with Legato
466 Networkers ".nsr" files where you can specify a lot of options on
467 how to handle files/directories (including denying further
468 parsing of .nsr files lower down into the directory trees). A
469 PDF version of the .nsr man page can be viewed at:
471 http://www.ifm.liu.se/~peter/nsr.pdf
473 Why: It's a nice alternative to "exclude" patterns for directories
474 which don't have regular pathnames. Also, it allows users to
475 control backup for themselves. Implementation should be pretty
476 simple. GNU tar >= 1.14 or so supports it, too.
478 Notes: I envision this as an optional feature to a fileset
481 Item 19: Implement new {Client}Run{Before|After}Job feature.
482 Date: 26 September 2005
483 Origin: Phil Stracchino <phil.stracchino at speakeasy dot net>
486 What: Some time ago, there was a discussion of RunAfterJob and
487 ClientRunAfterJob, and the fact that they do not run after failed
488 jobs. At the time, there was a suggestion to add a
489 RunAfterFailedJob directive (and, presumably, a matching
490 ClientRunAfterFailedJob directive), but to my knowledge these
491 were never implemented.
493 An alternate way of approaching the problem has just occurred to
494 me. Suppose the RunBeforeJob and RunAfterJob directives were
495 expanded in a manner something like this example:
498 Command = "/opt/bacula/etc/checkhost %c"
500 RunsAtJobLevels = All # All, Full, Diff, Inc
501 AbortJobOnError = Yes
504 Command = c:/bacula/systemstate.bat
506 RunsAtJobLevels = All # All, Full, Diff, Inc
511 Command = c:/bacula/deletestatefile.bat
513 RunsAtJobLevels = All # All, Full, Diff, Inc
518 Command = c:/bacula/somethingelse.bat
520 RunsAtJobLevels = All
525 Command = "/opt/bacula/etc/checkhost -v %c"
527 RunsAtJobLevels = All
533 Why: It would be a significant change to the structure of the
534 directives, but allows for a lot more flexibility, including
535 RunAfter commands that will run regardless of whether the job
536 succeeds, or RunBefore tasks that still allow the job to run even
537 if that specific RunBefore fails.
539 Notes: By Kern: I would prefer to have a single new Resource called
540 RunScript. More notes from Phil:
542 RunBeforeJob = yes|no
544 RunsAtJobLevels = All|Full|Diff|Inc
546 The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives
547 could be optional, and possibly RunsWhen as well.
549 AbortJobOnError would be ignored unless RunsWhen was set to Before
550 (or RunsBefore Job set to Yes), and would default to Yes if
551 omitted. If AbortJobOnError was set to No, failure of the script
552 would still generate a warning.
554 RunsOnSuccess would be ignored unless RunsWhen was set to After
555 (or RunsBeforeJob set to No), and default to Yes.
557 RunsOnFailure would be ignored unless RunsWhen was set to After,
560 Allow having the before/after status on the script command
561 line so that the same script can be used both before/after.
564 Item 20: Allow FD to initiate a backup
565 Origin: Frank Volf (frank at deze dot org)
566 Date: 17 November 2005
569 What: Provide some means, possibly by a restricted console that
570 allows a FD to initiate a backup, and that uses the connection
571 established by the FD to the Director for the backup so that
572 a Director that is firewalled can do the backup.
574 Why: Makes backup of laptops much easier.
576 Item 21: Multiple threads in file daemon for the same job
577 Date: 27 November 2005
578 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
581 What: I want the file daemon to start multiple threads for a backup
582 job so the fastest possible backup can be made.
584 The file daemon could parse the FileSet information and start
585 one thread for each File entry located on a separate
588 A configuration option in the job section should be used to
589 enable or disable this feature. The configuration option could
590 specify the maximum number of threads in the file daemon.
592 If the theads could spool the data to separate spool files
593 the restore process will not be much slower.
595 Why: Multiple concurrent backups of a large fileserver with many
596 disks and controllers will be much faster.
598 Notes: I am willing to try to implement this but I will probably
599 need some help and advice. (No problem -- Kern)
601 Item 22: Archival (removal) of User Files to Tape
605 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
608 What: The ability to archive data to storage based on certain parameters
609 such as age, size, or location. Once the data has been written to
610 storage and logged it is then pruned from the originating
611 filesystem. Note! We are talking about user's files and not
614 Why: This would allow fully automatic storage management which becomes
615 useful for large datastores. It would also allow for auto-staging
616 from one media type to another.
618 Example 1) Medical imaging needs to store large amounts of data.
619 They decide to keep data on their servers for 6 months and then put
620 it away for long term storage. The server then finds all files
621 older than 6 months writes them to tape. The files are then removed
624 Example 2) All data that hasn't been accessed in 2 months could be
625 moved from high-cost, fibre-channel disk storage to a low-cost
626 large-capacity SATA disk storage pool which doesn't have as quick of
627 access time. Then after another 6 months (or possibly as one
628 storage pool gets full) data is migrated to Tape.
631 Item 23: Deletion of Disk-Based Bacula Volumes
633 Origin: Ross Boylan <RossBoylan at stanfordalumni dot org> (edited
637 What: Provide a way for Bacula to automatically remove Volumes
638 from the filesystem, or optionally to truncate them.
639 Obviously, the Volume must be pruned prior removal.
641 Why: This would allow users more control over their Volumes and
642 prevent disk based volumes from consuming too much space.
644 Notes: The following two directives might do the trick:
646 Volume Data Retention = <time period>
647 Remove Volume After = <time period>
649 The migration project should also remove a Volume that is
650 migrated. This might also work for tape Volumes.
653 Item 24: Accurate restoration of renamed/deleted files from
654 Incremental/Differential backups
655 Date: 28 November 2005
656 Origin: Martin Simmons (martin at lispworks dot com)
659 What: When restoring a fileset for a specified date (including "most
660 recent"), Bacula should give you exactly the files and directories
661 that existed at the time of the last backup prior to that date.
663 Currently this only works if the last backup was a Full backup.
664 When the last backup was Incremental/Differential, files and
665 directories that have been renamed or deleted since the last Full
666 backup are not currently restored correctly. Ditto for files with
667 extra/fewer hard links than at the time of the last Full backup.
669 Why: Incremental/Differential would be much more useful if this worked.
671 Notes: Item 14 (Merging of multiple backups into a single one) seems to
672 rely on this working, otherwise the merged backups will not be
673 truely equivalent to a Full backup.
675 Kern: notes shortened. This can be done without the need for
676 inodes. It is essentially the same as the current Verify job,
677 but one additional database record must be written, which does
678 not need any database change.
680 Item 25: Implement creation and maintenance of copy pools
681 Date: 27 November 2005
682 Origin: David Boyes (dboyes at sinenomine dot net)
685 What: I would like Bacula to have the capability to write copies
686 of backed-up data on multiple physical volumes selected
687 from different pools without transferring the data
688 multiple times, and to accept any of the copy volumes
689 as valid for restore.
691 Why: In many cases, businesses are required to keep offsite
692 copies of backup volumes, or just wish for simple
693 protection against a human operator dropping a storage
694 volume and damaging it. The ability to generate multiple
695 volumes in the course of a single backup job allows
696 customers to simple check out one copy and send it
697 offsite, marking it as out of changer or otherwise
698 unavailable. Currently, the library and magazine
699 management capability in Bacula does not make this process
702 Restores would use the copy of the data on the first
703 available volume, in order of copy pool chain definition.
705 This is also a major scalability issue -- as the number of
706 clients increases beyond several thousand, and the volume
707 of data increases, transferring the data multiple times to
708 produce additional copies of the backups will become
709 physically impossible due to transfer speed
710 issues. Generating multiple copies at server side will
711 become the only practical option.
713 How: I suspect that this will require adding a multiplexing
714 SD that appears to be a SD to a specific FD, but 1-n FDs
715 to the specific back end SDs managing the primary and copy
716 pools. Storage pools will also need to acquire parameters
717 to define the pools to be used for copies.
719 Notes: I would commit some of my developers' time if we can agree
720 on the design and behavior.
723 ============= Empty Feature Request form ===========
724 Item n: One line summary ...
726 Origin: Name and email of originator.
729 What: More detailed explanation ...
731 Why: Why it is important ...
733 Notes: Additional notes or features (omit if not used)
734 ============== End Feature Request form ==============