3 Bacula Projects Roadmap
6 Below, you will find more information on future projects:
8 Item 1: Implement a Migration job type that will move the job
9 data from one device to another.
10 Origin: Sponsored by Riege Sofware International GmbH. Contact:
11 Daniel Holtkamp <holtkamp at riege dot com>
13 Status: Partially coded in 1.37 -- much more to do. Assigned to
16 What: The ability to copy, move, or archive data that is on a
17 device to another device is very important.
19 Why: An ISP might want to backup to disk, but after 30 days
20 migrate the data to tape backup and delete it from
21 disk. Bacula should be able to handle this
22 automatically. It needs to know what was put where,
23 and when, and what to migrate -- it is a bit like
24 retention periods. Doing so would allow space to be
25 freed up for current backups while maintaining older
28 Notes: Riege Software have asked for the following migration
31 Highwater mark (stopped by Lowwater mark?)
33 Notes: Migration could be additionally triggered by:
38 Item 2: Implement extraction of Win32 BackupWrite data.
39 Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
41 Status: Assigned to Thorsten. Implemented in current CVS
43 What: This provides the Bacula File daemon with code that
44 can pick apart the stream output that Microsoft writes
45 for BackupWrite data, and thus the data can be read
46 and restored on non-Win32 machines.
48 Why: BackupWrite data is the portable=no option in Win32
49 FileSets, and in previous Baculas, this data could
50 only be extracted using a Win32 FD. With this new code,
51 the Windows data can be extracted and restored on
55 Item 3: Implement a Bacula GUI/management tool using Python
62 What: Implement a Bacula console, and management tools
65 Why: Don't we already have a wxWidgets GUI? Yes, but
66 it is written in C++ and changes to the user interface
67 must be hand tailored using C++ code. By developing
68 the user interface using Qt designer, the interface
69 can be very easily updated and most of the new Python
70 code will be automatically created. The user interface
71 changes become very simple, and only the new features
72 must be implement. In addition, the code will be in
73 Python, which will give many more users easy (or easier)
74 access to making additions or modifications.
76 Item 4: Implement a Python interface to the Bacula catalog.
81 What: Implement an interface for Python scripts to access
82 the catalog through Bacula.
84 Why: This will permit users to customize Bacula through
87 Item 5: Implement more Python events in Bacula.
92 What: Allow Python scripts to be called at more places
93 within Bacula and provide additional access to Bacula
96 Why: This will permit users to customize Bacula through
104 Also add a way to get a listing of currently running
105 jobs (possibly also scheduled jobs).
108 Item 6: Implement Base jobs.
109 Date: 28 October 2005
113 What: A base job is sort of like a Full save except that you
114 will want the FileSet to contain only files that are
115 unlikely to change in the future (i.e. a snapshot of
116 most of your system after installing it). After the
117 base job has been run, when you are doing a Full save,
118 you specify one or more Base jobs to be used. All
119 files that have been backed up in the Base job/jobs but
120 not modified will then be excluded from the backup.
121 During a restore, the Base jobs will be automatically
122 pulled in where necessary.
124 Why: This is something none of the competition does, as far as
125 we know (except perhpas BackupPC, which is a Perl program that
126 saves to disk only). It is big win for the user, it
127 makes Bacula stand out as offering a unique
128 optimization that immediately saves time and money.
129 Basically, imagine that you have 100 nearly identical
130 Windows or Linux machine containing the OS and user
131 files. Now for the OS part, a Base job will be backed
132 up once, and rather than making 100 copies of the OS,
133 there will be only one. If one or more of the systems
134 have some files updated, no problem, they will be
135 automatically restored.
137 Notes: Huge savings in tape usage even for a single machine.
138 Will require more resources because the DIR must send
139 FD a list of files/attribs, and the FD must search the
140 list and compare it for each file to be saved.
142 Item 7: Add Plug-ins to the FileSet Include statements.
143 Date: 28 October 2005
145 Status: Partially coded in 1.37 -- much more to do.
147 What: Allow users to specify wild-card and/or regular
148 expressions to be matched in both the Include and
149 Exclude directives in a FileSet. At the same time,
150 allow users to define plug-ins to be called (based on
151 regular expression/wild-card matching).
153 Why: This would give the users the ultimate ability to control
154 how files are backed up/restored. A user could write a
155 plug-in knows how to backup his Oracle database without
156 stopping/starting it, for example.
158 Item 8: Implement huge exclude list support using hashing.
159 Date: 28 October 2005
163 What: Allow users to specify very large exclude list (currently
164 more than about 1000 files is too many).
166 Why: This would give the users the ability to exclude all
167 files that are loaded with the OS (e.g. using rpms
168 or debs). If the user can restore the base OS from
169 CDs, there is no need to backup all those files. A
170 complete restore would be to restore the base OS, then
171 do a Bacula restore. By excluding the base OS files, the
172 backup set will be *much* smaller.
175 Item 9: Implement data encryption (as opposed to communications
177 Date: 28 October 2005
178 Origin: Sponsored by Landon and 13 contributors to EFF.
179 Status: Landon Fuller is currently implementing this.
181 What: Currently the data that is stored on the Volume is not
182 encrypted. For confidentiality, encryption of data at
183 the File daemon level is essential.
184 Data encryption encrypts the data in the File daemon and
185 decrypts the data in the File daemon during a restore.
187 Why: Large sites require this.
189 Item 10: Permit multiple Media Types in an Autochanger
193 What: Modify the Storage daemon so that multiple Media Types
194 can be specified in an autochanger. This would be somewhat
195 of a simplistic implementation in that each drive would
196 still be allowed to have only one Media Type. However,
197 the Storage daemon will ensure that only a drive with
198 the Media Type that matches what the Director specifies
201 Why: This will permit user with several different drive types
202 to make full use of their autochangers.
204 Item 11: Allow two different autochanger definitions that refer
205 to the same autochanger.
206 Date: 28 October 2005
210 What: Currently, the autochanger script is locked based on
211 the autochanger. That is, if multiple drives are being
212 simultaneously used, the Storage daemon ensures that only
213 one drive at a time can access the mtx-changer script.
214 This change would base the locking on the control device,
215 rather than the autochanger. It would then permit two autochanger
216 definitions for the same autochanger, but with different
217 drives. Logically, the autochanger could then be "partitioned"
218 for different jobs, clients, or class of jobs, and if the locking
219 is based on the control device (e.g. /dev/sg0) the mtx-changer
220 script will be locked appropriately.
222 Why: This will permit users to partition autochangers for specific
223 use. It would also permit implementation of multiple Media
224 Types with no changes to the Storage daemon.
226 Item 12: Implement red/black binary tree routines.
227 Date: 28 October 2005
231 What: Implement a red/black binary tree class. This could
232 then replace the current binary insert/search routines
233 used in the restore in memory tree. This could significantly
234 speed up the creation of the in memory restore tree.
236 Why: Performance enhancement.
238 Item 13L Improve Baculas tape and drive usage management.
239 Date: 8 November 2005, November 11, 2005
240 Origin: Adam Thornton <athornton at sinenomine dot net>,
241 Arno Lehmann <al at its-lehmann dot de>
244 What: Make Bacula manage tape life cycle information, tape reuse
245 times and drive cleaning cycles.
247 Why: All three parts of this project are important when operating
249 We need to know which tapes need replacement, and we need to
250 make sure the drives are cleaned when necessary. While many
251 tape libraries and even autoloaders can handle all this
252 automatically, support by Bacula can be helpful for smaller
253 (older) libraries and single drives. Limiting the number of
254 times a tape is used might prevent tape errors when using
255 tapes until the drives can't read it any more. Also, checking
256 drive status during operation can prevent some failures (as I
257 [Arno] had to learn the hard way...)
259 Notes: First, Bacula could (and even does, to some limited extent)
260 record tape and drive usage. For tapes, the number of mounts,
261 the amount of data, and the time the tape has actually been
262 running could be recorded. Data fields for Read and Write
263 time and Number of mounts already exist in the catalog (I'm
264 not sure if VolBytes is the sum of all bytes ever written to
265 that volume by Bacula). This information can be important
266 when determining which media to replace. The ability to mark
267 Volumes as "used up" after a given number of write cycles
268 should also be implemented so that a tape is never actually
269 worn out. For the tape drives known to Bacula, similar
270 information is interesting to determine the device status and
271 expected life time: Time it's been Reading and Writing, number
272 of tape Loads / Unloads / Errors. This information is not yet
273 recorded as far as I [Arno] know. A new volume status would
274 be necessary for the new state, like "Used up" or "Worn out".
275 Volumes with this state could be used for restores, but not
276 for writing. These volumes should be migrated first (assuming
277 migration is implemented) and, once they are no longer needed,
278 could be moved to a Trash pool.
280 The next step would be to implement a drive cleaning setup.
281 Bacula already has knowledge about cleaning tapes. Once it
282 has some information about cleaning cycles (measured in drive
283 run time, number of tapes used, or calender days, for example)
284 it can automatically execute tape cleaning (with an
285 autochanger, obviously) or ask for operator assistence loading
288 The final step would be to implement TAPEALERT checks not only
289 when changing tapes and only sending the information to the
290 administrator, but rather checking after each tape error,
291 checking on a regular basis (for example after each tape
292 file), and also before unloading and after loading a new tape.
293 Then, depending on the drives TAPEALERT state and the known
294 drive cleaning state Bacula could automatically schedule later
295 cleaning, clean immediately, or inform the operator.
297 Implementing this would perhaps require another catalog change
298 and perhaps major changes in SD code and the DIR-SD protocoll,
299 so I'd only consider this worth implementing if it would
300 actually be used or even needed by many people.
302 Implemention of these projects could happen in three distinct
303 sub-projects: Measuring Tape and Drive usage, retiring
304 volumes, and handling drive cleaning and TAPEALERTs.
307 Item 14: Merging of multiple backups into a single one. (Also called Synthetic
308 Backup or Consolidation).
310 Origin: Marc Cousin and Eric Bollengier
311 Date: 15 November 2005
312 Status: Depends on first implementing project Item 1 (Migration).
314 What: A merged backup is a backup made without connecting to the Client.
315 It would be a Merge of existing backups into a single backup.
316 In effect, it is like a restore but to the backup medium.
318 For instance, say that last sunday we made a full backup. Then
319 all week long, we created incremental backups, in order to do
320 them fast. Now comes sunday again, and we need another full.
321 The merged backup makes it possible to do instead an incremental
322 backup (during the night for instance), and then create a merged
323 backup during the day, by using the full and incrementals from
324 the week. The merged backup will be exactly like a full made
325 sunday night on the tape, but the production interruption on the
326 Client will be minimal, as the Client will only have to send
329 In fact, if it's done correctly, you could merge all the
330 Incrementals into single Incremental, or all the Incrementals
331 and the last Differential into a new Differential, or the Full,
332 last differential and all the Incrementals into a new Full
333 backup. And there is no need to involve the Client.
335 Why: The benefit is that :
336 - the Client just does an incremental ;
337 - the merged backup on tape is just as a single full backup,
338 and can be restored very fast.
340 This is also a way of reducing the backup data since the old
341 data can then be pruned (or not) from the catalog, possibly
342 allowing older volumes to be recycled
344 Item 15: Automatic disabling of devices
346 Origin: Peter Eriksson <peter at ifm.liu dot se>
349 What: After a configurable amount of fatal errors with a tape drive
350 Bacula should automatically disable further use of a certain
351 tape drive. There should also be "disable"/"enable" commands in
354 Why: On a multi-drive jukebox there is a possibility of tape drives
355 going bad during large backups (needing a cleaning tape run,
356 tapes getting stuck). It would be advantageous if Bacula would
357 automatically disable further use of a problematic tape drive
358 after a configurable amount of errors has occured.
360 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
361 where tapes occasionally get stuck inside the drive. Bacula will
362 notice that the "mtx-changer" command will fail and then fail
363 any backup jobs trying to use that drive. However, it will still
364 keep on trying to run new jobs using that drive and fail -
365 forever, and thus failing lots and lots of jobs... Since we have
366 many drives Bacula could have just automatically disabled
367 further use of that drive and used one of the other ones
371 Item 16: Directive/mode to backup only file changes, not entire file
372 Date: 11 November 2005
373 Origin: Joshua Kugler <joshua dot kugler at uaf dot edu>
374 Marek Bajon <mbajon at bimsplus dot com dot pl>
377 What: Currently when a file changes, the entire file will be backed up in
378 the next incremental or full backup. To save space on the tapes
379 it would be nice to have a mode whereby only the changes to the
380 file would be backed up when it is changed.
382 Why: This would save lots of space when backing up large files such as
383 logs, mbox files, Outlook PST files and the like.
385 Notes: This would require the usage of disk-based volumes as comparing
386 files would not be feasible using a tape drive.
388 Item 17: Quick release of FD-SD connection
389 Origin: Frank Volf (frank at deze dot org)
390 Date: 17 november 2005
393 What: In the bacula implementation a backup is finished after all data
394 and attributes are succesfully written to storage. When using a
395 tape backup it is very annoying that a backup can take a day,
396 simply because the current tape (or whatever) is full and the
397 administrator has not put a new one in. During that time the
398 system cannot be taken off-line, because there is still an open
399 session between the storage daemon and the file daemon on the
402 Although this is a very good strategey for making "safe backups"
403 This can be annoying for e.g. laptops, that must remain
404 connected until the bacukp is completed.
406 Using a new feature called "migration" it will be possible to
407 spool first to harddisk (using a special 'spool' migration
408 scheme) and then migrate the backup to tape.
410 There is still the problem of getting the attributes committed.
411 If it takes a very long time to do, with the current code, the
412 job has not terminated, and the File daemon is not freed up. The
413 Storage daemon should release the File daemon as soon as all the
414 file data and all the attributes have been sent to it (the SD).
415 Currently the SD waits until everything is on tape and all the
416 attributes are transmitted to the Director before signalling
417 completion to the FD. I don't think I would have any problem
418 changing this. The reason is that even if the FD reports back to
419 the Dir that all is OK, the job will not terminate until the SD
420 has done the same thing -- so in a way keeping the SD-FD link
421 open to the very end is not really very productive ...
423 Why: Makes backup of laptops much easier.
426 Item 18: Add support for CACHEDIR.TAG
427 Origin: Norbert Kiesel <nkiesel at tbdnetworks dot com>
428 Date: 21 November 2005
431 What: CACHDIR.TAG is a proposal for identifying directories which
432 should be ignored for archiving/backup. It works by ignoring
433 directory trees which have a file named CACHEDIR.TAG with a
434 specific content. See
435 http://www.brynosaurus.com/cachedir/spec.html
439 I suggest that if this is implemented (I've also asked for this
440 feature some year ago) that it is made compatible with Legato
441 Networkers ".nsr" files where you can specify a lot of options on
442 how to handle files/directories (including denying further
443 parsing of .nsr files lower down into the directory trees). A
444 PDF version of the .nsr man page can be viewed at:
446 http://www.ifm.liu.se/~peter/nsr.pdf
448 Why: It's a nice alternative to "exclude" patterns for directories
449 which don't have regular pathnames. Also, it allows users to
450 control backup for themself. Implementation should be pretty
451 simple. GNU tar >= 1.14 or so supports it, too.
453 Notes: I envision this as an optional feature to a fileset
456 Item 19: Implement new {Client}Run{Before|After}Job feature.
457 Date: 26 September 2005
458 Origin: Phil Stracchino <phil.stracchino at speakeasy dot net>
461 What: Some time ago, there was a discussion of RunAfterJob and
462 ClientRunAfterJob, and the fact that they do not run after failed
463 jobs. At the time, there was a suggestion to add a
464 RunAfterFailedJob directive (and, presumably, a matching
465 ClientRunAfterFailedJob directive), but to my knowledge these
466 were never implemented.
468 An alternate way of approaching the problem has just occurred to
469 me. Suppose the RunBeforeJob and RunAfterJob directives were
470 expanded in a manner something like this example:
473 Command = "/opt/bacula/etc/checkhost %c"
475 RunsAtJobLevels = All # All, Full, Diff, Inc
476 AbortJobOnError = Yes
479 Command = c:/bacula/systemstate.bat
481 RunsAtJobLevels = All # All, Full, Diff, Inc
486 Command = c:/bacula/deletestatefile.bat
488 RunsAtJobLevels = All # All, Full, Diff, Inc
493 Command = c:/bacula/somethingelse.bat
495 RunsAtJobLevels = All
500 Command = "/opt/bacula/etc/checkhost -v %c"
502 RunsAtJobLevels = All
508 Why: It would be a significant change to the structure of the
509 directives, but allows for a lot more flexibility, including
510 RunAfter commands that will run regardless of whether the job
511 succeeds, or RunBefore tasks that still allow the job to run even
512 if that specific RunBefore fails.
514 Notes: By Kern: I would prefer to have a single new Resource called
515 RunScript. More notes from Phil:
517 RunBeforeJob = yes|no
519 RunsAtJobLevels = All|Full|Diff|Inc
521 The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives
522 could be optional, and possibly RunsWhen as well.
524 AbortJobOnError would be ignored unless RunsWhen was set to Before
525 (or RunsBefore Job set to Yes), and would default to Yes if
526 omitted. If AbortJobOnError was set to No, failure of the script
527 would still generate a warning.
529 RunsOnSuccess would be ignored unless RunsWhen was set to After
530 (or RunsBeforeJob set to No), and default to Yes.
532 RunsOnFailure would be ignored unless RunsWhen was set to After,
535 Allow having the before/after status on the script command
536 line so that the same script can be used both before/after.
539 Item 20: Allow FD to initiate a backup
540 Origin: Frank Volf (frank at deze dot org)
541 Date: 17 november 2005
544 What: Provide some means, possibly by a restricted console that
545 allows a FD to initiate a backup, and that uses the connection
546 established by the FD to the Director for the backup so that
547 a Director that is firewalled can do the backup.
549 Why: Makes backup of laptops much easier.
552 ============= Empty Feature Request form ===========
553 Item n: One line summary ...
555 Origin: Name and email of originator.
558 What: More detailed explanation ...
560 Why: Why it is important ...
562 Notes: Additional notes or features (omit if not used)
563 ============== End Feature Request form ==============
projects / bacula / bacula / blob