3 Bacula Projects Roadmap
6 Below, you will find more information on future projects:
8 Item 1: Implement a Migration job type that will move the job
9 data from one device to another.
10 Origin: Sponsored by Riege Sofware International GmbH. Contact:
11 Daniel Holtkamp <holtkamp at riege dot com>
13 Status: Partially coded in 1.37 -- much more to do. Assigned to
16 What: The ability to copy, move, or archive data that is on a
17 device to another device is very important.
19 Why: An ISP might want to backup to disk, but after 30 days
20 migrate the data to tape backup and delete it from
21 disk. Bacula should be able to handle this
22 automatically. It needs to know what was put where,
23 and when, and what to migrate -- it is a bit like
24 retention periods. Doing so would allow space to be
25 freed up for current backups while maintaining older
28 Notes: Riege Software have asked for the following migration
31 Highwater mark (stopped by Lowwater mark?)
33 Notes: Migration could be additionally triggered by:
38 Item 2: Implement extraction of Win32 BackupWrite data.
39 Origin: Thorsten Engel <thorsten.engel at matrix-computer dot com>
41 Status: Assigned to Thorsten. Implemented in current CVS
43 What: This provides the Bacula File daemon with code that
44 can pick apart the stream output that Microsoft writes
45 for BackupWrite data, and thus the data can be read
46 and restored on non-Win32 machines.
48 Why: BackupWrite data is the portable=no option in Win32
49 FileSets, and in previous Baculas, this data could
50 only be extracted using a Win32 FD. With this new code,
51 the Windows data can be extracted and restored on
55 Item 3: Implement a Bacula GUI/management tool using Python
62 What: Implement a Bacula console, and management tools
65 Why: Don't we already have a wxWidgets GUI? Yes, but
66 it is written in C++ and changes to the user interface
67 must be hand tailored using C++ code. By developing
68 the user interface using Qt designer, the interface
69 can be very easily updated and most of the new Python
70 code will be automatically created. The user interface
71 changes become very simple, and only the new features
72 must be implement. In addition, the code will be in
73 Python, which will give many more users easy (or easier)
74 access to making additions or modifications.
76 Item 4: Implement a Python interface to the Bacula catalog.
81 What: Implement an interface for Python scripts to access
82 the catalog through Bacula.
84 Why: This will permit users to customize Bacula through
87 Item 5: Implement more Python events in Bacula.
92 What: Allow Python scripts to be called at more places
93 within Bacula and provide additional access to Bacula
96 Why: This will permit users to customize Bacula through
105 Item 6: Implement Base jobs.
106 Date: 28 October 2005
110 What: A base job is sort of like a Full save except that you
111 will want the FileSet to contain only files that are
112 unlikely to change in the future (i.e. a snapshot of
113 most of your system after installing it). After the
114 base job has been run, when you are doing a Full save,
115 you specify one or more Base jobs to be used. All
116 files that have been backed up in the Base job/jobs but
117 not modified will then be excluded from the backup.
118 During a restore, the Base jobs will be automatically
119 pulled in where necessary.
121 Why: This is something none of the competition does, as far as
122 we know (except perhpas BackupPC, which is a Perl program that
123 saves to disk only). It is big win for the user, it
124 makes Bacula stand out as offering a unique
125 optimization that immediately saves time and money.
126 Basically, imagine that you have 100 nearly identical
127 Windows or Linux machine containing the OS and user
128 files. Now for the OS part, a Base job will be backed
129 up once, and rather than making 100 copies of the OS,
130 there will be only one. If one or more of the systems
131 have some files updated, no problem, they will be
132 automatically restored.
134 Notes: Huge savings in tape usage even for a single machine.
135 Will require more resources because the DIR must send
136 FD a list of files/attribs, and the FD must search the
137 list and compare it for each file to be saved.
139 Item 7: Add Plug-ins to the FileSet Include statements.
140 Date: 28 October 2005
142 Status: Partially coded in 1.37 -- much more to do.
144 What: Allow users to specify wild-card and/or regular
145 expressions to be matched in both the Include and
146 Exclude directives in a FileSet. At the same time,
147 allow users to define plug-ins to be called (based on
148 regular expression/wild-card matching).
150 Why: This would give the users the ultimate ability to control
151 how files are backed up/restored. A user could write a
152 plug-in knows how to backup his Oracle database without
153 stopping/starting it, for example.
155 Item 8: Implement huge exclude list support using hashing.
156 Date: 28 October 2005
160 What: Allow users to specify very large exclude list (currently
161 more than about 1000 files is too many).
163 Why: This would give the users the ability to exclude all
164 files that are loaded with the OS (e.g. using rpms
165 or debs). If the user can restore the base OS from
166 CDs, there is no need to backup all those files. A
167 complete restore would be to restore the base OS, then
168 do a Bacula restore. By excluding the base OS files, the
169 backup set will be *much* smaller.
172 Item 9: Implement data encryption (as opposed to communications
174 Date: 28 October 2005
175 Origin: Sponsored by Landon and 13 contributors to EFF.
176 Status: Landon Fuller is currently implementing this.
178 What: Currently the data that is stored on the Volume is not
179 encrypted. For confidentiality, encryption of data at
180 the File daemon level is essential.
181 Data encryption encrypts the data in the File daemon and
182 decrypts the data in the File daemon during a restore.
184 Why: Large sites require this.
186 Item 10: Permit multiple Media Types in an Autochanger
190 What: Modify the Storage daemon so that multiple Media Types
191 can be specified in an autochanger. This would be somewhat
192 of a simplistic implementation in that each drive would
193 still be allowed to have only one Media Type. However,
194 the Storage daemon will ensure that only a drive with
195 the Media Type that matches what the Director specifies
198 Why: This will permit user with several different drive types
199 to make full use of their autochangers.
201 Item 11: Allow two different autochanger definitions that refer
202 to the same autochanger.
203 Date: 28 October 2005
207 What: Currently, the autochanger script is locked based on
208 the autochanger. That is, if multiple drives are being
209 simultaneously used, the Storage daemon ensures that only
210 one drive at a time can access the mtx-changer script.
211 This change would base the locking on the control device,
212 rather than the autochanger. It would then permit two autochanger
213 definitions for the same autochanger, but with different
214 drives. Logically, the autochanger could then be "partitioned"
215 for different jobs, clients, or class of jobs, and if the locking
216 is based on the control device (e.g. /dev/sg0) the mtx-changer
217 script will be locked appropriately.
219 Why: This will permit users to partition autochangers for specific
220 use. It would also permit implementation of multiple Media
221 Types with no changes to the Storage daemon.
223 Item 12: Implement red/black binary tree routines.
224 Date: 28 October 2005
228 What: Implement a red/black binary tree class. This could
229 then replace the current binary insert/search routines
230 used in the restore in memory tree. This could significantly
231 speed up the creation of the in memory restore tree.
233 Why: Performance enhancement.
235 Item 13: Let Bacula log tape usage and handle drive cleaning cycles.
236 Date: November 11, 2005
237 Origin: Arno Lehmann <al at its-lehmann dot de>
240 What: Make Bacula manage tape life cycle information and drive
243 Why: Both parts of this project are important when operating backups.
244 We need to know which tapes need replacement, and we need to
245 make sure the drives are cleaned when necessary. While many
246 tape libraries and even autoloaders can handle all this
247 automatically, support by Bacula can be helpful for smaller
248 (older) libraries and single drives. Also, checking drive
249 status during operation can prevent some failures (as I had to
250 learn the hard way...)
252 Notes: First, Bacula could (and even does, to some limited extent)
253 record tape and drive usage. For tapes, the number of mounts,
254 the amount of data, and the time the tape has actually been
255 running could be recorded. Data fields for Read and Write time
256 and Nmber of mounts already exist in the catalog (I'm not sure
257 if VolBytes is the sum of all bytes ever written to that volume
258 by Bacula). This information can be important when determining
259 which media to replace. For the tape drives known to Bacula,
260 similar information is interesting to determine the device
261 status and expected life time: Time it's been Reading and
262 Writing, number of tape Loads / Unloads / Errors. This
263 information is not yet recorded as far as I know.
265 The next step would be implementing drive cleaning setup.
266 Bacula already has knowledge about cleaning tapes. Once it has
267 some information about cleaning cycles (measured in drive run
268 time, number of tapes used, or calender days, for example) it
269 can automatically execute tape cleaning (with an autochanger,
270 obviously) or ask for operator assistence loading a cleaning
273 The next step would be to implement TAPEALERT checks not only
274 when changing tapes and only sending he information to the
275 administrator, but rather checking after each tape error,
276 checking on a regular basis (for example after each tape file),
277 and also before unloading and after loading a new tape. Then,
278 depending on the drives TAPEALERT state and the know drive
279 cleaning state Bacula could automatically schedule later
280 cleaning, clean immediately, or inform the operator.
282 Implementing this would perhaps require another catalog change
283 and perhaps major changes in SD code and the DIR-SD protocoll,
284 so I'd only consider this worth implementing if it would
285 actually be used or even needed by many people.
287 Item 14: Merging of multiple backups into a single one. (Also called Synthetic
288 Backup or Consolidation).
290 Origin: Marc Cousin and Eric Bollengier
291 Date: 15 November 2005
292 Status: Depends on first implementing project Item 1 (Migration).
294 What: A merged backup is a backup made without connecting to the Client.
295 It would be a Merge of existing backups into a single backup.
296 In effect, it is like a restore but to the backup medium.
298 For instance, say that last sunday we made a full backup. Then
299 all week long, we created incremental backups, in order to do
300 them fast. Now comes sunday again, and we need another full.
301 The merged backup makes it possible to do instead an incremental
302 backup (during the night for instance), and then create a merged
303 backup during the day, by using the full and incrementals from
304 the week. The merged backup will be exactly like a full made
305 sunday night on the tape, but the production interruption on the
306 Client will be minimal, as the Client will only have to send
309 In fact, if it's done correctly, you could merge all the
310 Incrementals into single Incremental, or all the Incrementals
311 and the last Differential into a new Differential, or the Full,
312 last differential and all the Incrementals into a new Full
313 backup. And there is no need to involve the Client.
315 Why: The benefit is that :
316 - the Client just does an incremental ;
317 - the merged backup on tape is just as a single full backup,
318 and can be restored very fast.
320 This is also a way of reducing the backup data since the old
321 data can then be pruned (or not) from the catalog, possibly
322 allowing older volumes to be recycled
324 Item 15: Automatic disabling of devices
326 Origin: Peter Eriksson <peter at ifm.liu dot se>
329 What: After a configurable amount of fatal errors with a tape drive
330 Bacula should automatically disable further use of a certain
331 tape drive. There should also be "disable"/"enable" commands in
334 Why: On a multi-drive jukebox there is a possibility of tape drives
335 going bad during large backups (needing a cleaning tape run,
336 tapes getting stuck). It would be advantageous if Bacula would
337 automatically disable further use of a problematic tape drive
338 after a configurable amount of errors has occured.
340 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
341 where tapes occasionally get stuck inside the drive. Bacula will
342 notice that the "mtx-changer" command will fail and then fail
343 any backup jobs trying to use that drive. However, it will still
344 keep on trying to run new jobs using that drive and fail -
345 forever, and thus failing lots and lots of jobs... Since we have
346 many drives Bacula could have just automatically disabled
347 further use of that drive and used one of the other ones
351 Item 16: Directive/mode to backup only file changes, not entire file
352 Date: 11 November 2005
353 Origin: Joshua Kugler <joshua dot kugler at uaf dot edu>
354 Marek Bajon <mbajon at bimsplus dot com dot pl>
357 What: Currently when a file changes, the entire file will be backed up in
358 the next incremental or full backup. To save space on the tapes
359 it would be nice to have a mode whereby only the changes to the
360 file would be backed up when it is changed.
362 Why: This would save lots of space when backing up large files such as
363 logs, mbox files, Outlook PST files and the like.
365 Notes: This would require the usage of disk-based volumes as comparing
366 files would not be feasible using a tape drive.
368 Item 17: Quick release of FD-SD connection
369 Origin: Frank Volf (frank at deze dot org)
370 Date: 17 november 2005
373 What: In the bacula implementation a backup is finished after all data
374 and attributes are succesfully written to storage. When using a
375 tape backup it is very annoying that a backup can take a day,
376 simply because the current tape (or whatever) is full and the
377 administrator has not put a new one in. During that time the
378 system cannot be taken off-line, because there is still an open
379 session between the storage daemon and the file daemon on the
382 Although this is a very good strategey for making "safe backups"
383 This can be annoying for e.g. laptops, that must remain
384 connected until the bacukp is completed.
386 Using a new feature called "migration" it will be possible to
387 spool first to harddisk (using a special 'spool' migration
388 scheme) and then migrate the backup to tape.
390 There is still the problem of getting the attributes committed.
391 If it takes a very long time to do, with the current code, the
392 job has not terminated, and the File daemon is not freed up. The
393 Storage daemon should release the File daemon as soon as all the
394 file data and all the attributes have been sent to it (the SD).
395 Currently the SD waits until everything is on tape and all the
396 attributes are transmitted to the Director before signalling
397 completion to the FD. I don't think I would have any problem
398 changing this. The reason is that even if the FD reports back to
399 the Dir that all is OK, the job will not terminate until the SD
400 has done the same thing -- so in a way keeping the SD-FD link
401 open to the very end is not really very productive ...
403 Why: Makes backup of laptops much easier.
406 Item 18: Add support for CACHEDIR.TAG
407 Origin: Norbert Kiesel <nkiesel at tbdnetworks dot com>
408 Date: 21 November 2005
411 What: CACHDIR.TAG is a proposal for identifying directories which
412 should be ignored for archiving/backup. It works by ignoring
413 directory trees which have a file named CACHEDIR.TAG with a
414 specific content. See
415 http://www.brynosaurus.com/cachedir/spec.html
419 I suggest that if this is implemented (I've also asked for this
420 feature some year ago) that it is made compatible with Legato
421 Networkers ".nsr" files where you can specify a lot of options on
422 how to handle files/directories (including denying further
423 parsing of .nsr files lower down into the directory trees). A
424 PDF version of the .nsr man page can be viewed at:
426 http://www.ifm.liu.se/~peter/nsr.pdf
428 Why: It's a nice alternative to "exclude" patterns for directories
429 which don't have regular pathnames. Also, it allows users to
430 control backup for themself. Implementation should be pretty
431 simple. GNU tar >= 1.14 or so supports it, too.
433 Notes: I envision this as an optional feature to a fileset
436 Item 19: Implement new {Client}Run{Before|After}Job feature.
437 Date: 26 September 2005
438 Origin: Phil Stracchino <phil.stracchino at speakeasy dot net>
441 What: Some time ago, there was a discussion of RunAfterJob and
442 ClientRunAfterJob, and the fact that they do not run after failed
443 jobs. At the time, there was a suggestion to add a
444 RunAfterFailedJob directive (and, presumably, a matching
445 ClientRunAfterFailedJob directive), but to my knowledge these
446 were never implemented.
448 An alternate way of approaching the problem has just occurred to
449 me. Suppose the RunBeforeJob and RunAfterJob directives were
450 expanded in a manner something like this example:
453 Command = "/opt/bacula/etc/checkhost %c"
455 RunsAtJobLevels = All # All, Full, Diff, Inc
456 AbortJobOnError = Yes
459 Command = c:/bacula/systemstate.bat
461 RunsAtJobLevels = All # All, Full, Diff, Inc
466 Command = c:/bacula/deletestatefile.bat
468 RunsAtJobLevels = All # All, Full, Diff, Inc
473 Command = c:/bacula/somethingelse.bat
475 RunsAtJobLevels = All
480 Command = "/opt/bacula/etc/checkhost -v %c"
482 RunsAtJobLevels = All
488 Why: It would be a significant change to the structure of the
489 directives, but allows for a lot more flexibility, including
490 RunAfter commands that will run regardless of whether the job
491 succeeds, or RunBefore tasks that still allow the job to run even
492 if that specific RunBefore fails.
494 Notes: By Kern: I would prefer to have a single new Resource called
495 RunScript. More notes from Phil:
497 RunBeforeJob = yes|no
499 RunsAtJobLevels = All|Full|Diff|Inc
501 The AbortJobOnError, RunsOnSuccess and RunsOnFailure directives
502 could be optional, and possibly RunsWhen as well.
504 AbortJobOnError would be ignored unless RunsWhen was set to Before
505 (or RunsBefore Job set to Yes), and would default to Yes if
506 omitted. If AbortJobOnError was set to No, failure of the script
507 would still generate a warning.
509 RunsOnSuccess would be ignored unless RunsWhen was set to After
510 (or RunsBeforeJob set to No), and default to Yes.
512 RunsOnFailure would be ignored unless RunsWhen was set to After,
515 Allow having the before/after status on the script command
516 line so that the same script can be used both before/after.
519 Item 20: "Maximum Rewrite (Recycle) Times" for a tape
520 Date: 8 November 2005
521 Origin: Adam Thornton <athornton at sinenomine dot net>
524 What: The ability to use a Volume for at most N re-write times, marking
525 it unavailable after that.
527 Why: I was working with a customer this morning who mentioned that it
528 would be useful to automatically age out tapes that had been
529 rewritten enough times that the media lifespan was becoming
530 questionable (his old backup system supported saying "rewrite this
531 tape 50 times at most." Neither Maximum Volume Jobs nor Volume Use
532 Duration quite does this, because you can have multiple jobs per
533 volume, and it's not entirely guaranteed that time and number of
534 write cycles map linearly (although they probably do in actual
535 usage), and also because "Used" volumes can be recycled, and the
536 status we want here is something like "full and now unwriteable."
537 I haven't looked but I suspect this would require a change in the
538 database format, to keep track of how many cycles a volume has
539 been through, and maybe an additional Status type.
542 Item 21: Allow FD to initiate a backup
543 Origin: Frank Volf (frank at deze dot org)
544 Date: 17 november 2005
547 What: Provide some means, possibly by a restricted console that
548 allows a FD to initiate a backup, and that uses the connection
549 established by the FD to the Director for the backup so that
550 a Director that is firewalled can do the backup.
552 Why: Makes backup of laptops much easier.
555 ============= Empty Feature Request form ===========
556 Item n: One line summary ...
558 Origin: Name and email of originator.
561 What: More detailed explanation ...
563 Why: Why it is important ...
565 Notes: Additional notes or features (omit if not used)
566 ============== End Feature Request form ==============
projects / bacula / bacula / blob