3 Bacula Projects Roadmap
4 Status updated 04 Jun 2009
8 Item 1: Ability to restart failed jobs
9 Item 2: 'restore' menu: enter a JobId, automatically select dependents
10 Item 3: Scheduling syntax that permits more flexibility and options
11 Item 4: Data encryption on storage daemon
12 Item 5: Deletion of disk Volumes when pruned
13 Item 6: Implement Base jobs
14 Item 7: Add ability to Verify any specified Job.
15 Item 8: Improve Bacula's tape and drive usage and cleaning management
16 Item 9: Allow FD to initiate a backup
17 Item 10: Restore from volumes on multiple storage daemons
18 Item 11: Implement Storage daemon compression
19 Item 12: Reduction of communications bandwidth for a backup
20 Item 13: Ability to reconnect a disconnected comm line
21 Item 14: Start spooling even when waiting on tape
22 Item 15: Enable/disable compression depending on storage device (disk/tape)
23 Item 16: Include all conf files in specified directory
24 Item 17: Multiple threads in file daemon for the same job
25 Item 18: Possibilty to schedule Jobs on last Friday of the month
26 Item 19: Include timestamp of job launch in "stat clients" output
27 Item 20: Cause daemons to use a specific IP address to source communications
28 Item 21: Message mailing based on backup types
29 Item 22: Ability to import/export Bacula database entities
30 Item 23: "Maximum Concurrent Jobs" for drives when used with changer device
31 Item 24: Implementation of running Job speed limit.
32 Item 25: Add an override in Schedule for Pools based on backup types
33 Item 26: Automatic promotion of backup levels based on backup size
34 Item 27: Allow inclusion/exclusion of files in a fileset by creation/mod times
35 Item 28: Archival (removal) of User Files to Tape
36 Item 29: An option to operate on all pools with update vol parameters
37 Item 30: Automatic disabling of devices
38 Item 31: List InChanger flag when doing restore.
39 Item 32: Ability to defer Batch Insert to a later time
40 Item 33: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
41 Item 34: Enable persistent naming/number of SQL queries
42 Item 35: Port bat to Win32
43 Item 36: Bacula Dir, FD and SD to support proxies
44 Item 37: Add Minumum Spool Size directive
45 Item 38: Backup and Restore of Windows Encrypted Files using Win raw encryption
46 Item 39: Implement an interface between Bacula and Amazon's S3.
47 Item 40: Convert Bacula existing tray monitor on Windows to a stand alone program
49 Item 1: Ability to restart failed jobs
54 What: Often jobs fail because of a communications line drop or max run time,
55 cancel, or some other non-critical problem. Currrently any data
56 saved is lost. This implementation should modify the Storage daemon
57 so that it saves all the files that it knows are completely backed
60 The jobs should then be marked as incomplete and a subsequent
61 Incremental Accurate backup will then take into account all the
64 Why: Avoids backuping data already saved.
66 Notes: Requires Accurate to restart correctly. Must completed have a minimum
67 volume of data or files stored on Volume before enabling.
70 Item 2: 'restore' menu: enter a JobId, automatically select dependents
71 Origin: Graham Keeling (graham@equiinet.com)
76 What: Add to the bconsole 'restore' menu the ability to select a job
77 by JobId, and have bacula automatically select all the dependent jobs.
79 Why: Currently, you either have to...
80 a) laboriously type in a date that is greater than the date of the backup that
81 you want and is less than the subsequent backup (bacula then figures out the
83 b) manually figure out all the JobIds that you want and laboriously type them
85 It would be extremely useful (in a programmatical sense, as well as for humans)
86 to be able to just give it a single JobId and let bacula do the hard work (work
87 that it already knows how to do).
89 Notes (Kern): I think this should either be modified to have Bacula print
90 a list of dates that the user can choose from as is done in bwx-console and
91 bat or the name of this command must be carefully chosen so that the user
92 clearly understands that the JobId is being used to specify what Job and the
93 date to which he wishes the restore to happen.
96 Item 3: Scheduling syntax that permits more flexibility and options
97 Date: 15 December 2006
98 Origin: Gregory Brauer (greg at wildbrain dot com) and
99 Florian Schnabel <florian.schnabel at docufy dot de>
102 What: Currently, Bacula only understands how to deal with weeks of the
103 month or weeks of the year in schedules. This makes it impossible
104 to do a true weekly rotation of tapes. There will always be a
105 discontinuity that will require disruptive manual intervention at
106 least monthly or yearly because week boundaries never align with
107 month or year boundaries.
109 A solution would be to add a new syntax that defines (at least)
110 a start timestamp, and repetition period.
112 An easy option to skip a certain job on a certain date.
115 Why: Rotated backups done at weekly intervals are useful, and Bacula
116 cannot currently do them without extensive hacking.
118 You could then easily skip tape backups on holidays. Especially
119 if you got no autochanger and can only fit one backup on a tape
120 that would be really handy, other jobs could proceed normally
121 and you won't get errors that way.
124 Notes: Here is an example syntax showing a 3-week rotation where full
125 Backups would be performed every week on Saturday, and an
126 incremental would be performed every week on Tuesday. Each
127 set of tapes could be removed from the loader for the following
128 two cycles before coming back and being reused on the third
129 week. Since the execution times are determined by intervals
130 from a given point in time, there will never be any issues with
131 having to adjust to any sort of arbitrary time boundary. In
132 the example provided, I even define the starting schedule
133 as crossing both a year and a month boundary, but the run times
134 would be based on the "Repeat" value and would therefore happen
139 Name = "Week 1 Rotation"
140 #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
144 Start = 2006-12-30 01:00
148 #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
152 Start = 2007-01-02 01:00
159 Name = "Week 2 Rotation"
160 #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
164 Start = 2007-01-06 01:00
168 #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
172 Start = 2007-01-09 01:00
179 Name = "Week 3 Rotation"
180 #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
184 Start = 2007-01-13 01:00
188 #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
192 Start = 2007-01-16 01:00
198 Notes: Kern: I have merged the previously separate project of skipping
199 jobs (via Schedule syntax) into this.
202 Item 4: Data encryption on storage daemon
203 Origin: Tobias Barth <tobias.barth at web-arts.com>
204 Date: 04 February 2009
207 What: The storage demon should be able to do the data encryption that can currently be done by the file daemon.
209 Why: This would have 2 advantages: 1) one could encrypt the data of unencrypted tapes by doing a migration job, and 2) the storage daemon would be the only machine that would have to keep the encryption keys.
212 As an addendum to the feature request, here are some crypto
213 implementation details I wrote up regarding SD-encryption back in Jan
215 http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html
218 Item 5: Deletion of disk Volumes when pruned
220 Origin: Ross Boylan <RossBoylan at stanfordalumni dot org> (edited
224 What: Provide a way for Bacula to automatically remove Volumes
225 from the filesystem, or optionally to truncate them.
226 Obviously, the Volume must be pruned prior removal.
228 Why: This would allow users more control over their Volumes and
229 prevent disk based volumes from consuming too much space.
231 Notes: The following two directives might do the trick:
233 Volume Data Retention = <time period>
234 Remove Volume After = <time period>
236 The migration project should also remove a Volume that is
237 migrated. This might also work for tape Volumes.
239 Notes: (Kern). The data fields to control this have been added
240 to the new 3.0.0 database table structure.
243 Item 6: Implement Base jobs
244 Date: 28 October 2005
248 What: A base job is sort of like a Full save except that you
249 will want the FileSet to contain only files that are
250 unlikely to change in the future (i.e. a snapshot of
251 most of your system after installing it). After the
252 base job has been run, when you are doing a Full save,
253 you specify one or more Base jobs to be used. All
254 files that have been backed up in the Base job/jobs but
255 not modified will then be excluded from the backup.
256 During a restore, the Base jobs will be automatically
257 pulled in where necessary.
259 Why: This is something none of the competition does, as far as
260 we know (except perhaps BackupPC, which is a Perl program that
261 saves to disk only). It is big win for the user, it
262 makes Bacula stand out as offering a unique
263 optimization that immediately saves time and money.
264 Basically, imagine that you have 100 nearly identical
265 Windows or Linux machine containing the OS and user
266 files. Now for the OS part, a Base job will be backed
267 up once, and rather than making 100 copies of the OS,
268 there will be only one. If one or more of the systems
269 have some files updated, no problem, they will be
270 automatically restored.
272 Notes: Huge savings in tape usage even for a single machine.
273 Will require more resources because the DIR must send
274 FD a list of files/attribs, and the FD must search the
275 list and compare it for each file to be saved.
278 Item 7: Add ability to Verify any specified Job.
279 Date: 17 January 2008
280 Origin: portrix.net Hamburg, Germany.
281 Contact: Christian Sabelmann
282 Status: 70% of the required Code is part of the Verify function since v. 2.x
285 The ability to tell Bacula which Job should verify instead of
286 automatically verify just the last one.
289 It is sad that such a powerfull feature like Verify Jobs
290 (VolumeToCatalog) is restricted to be used only with the last backup Job
291 of a client. Actual users who have to do daily Backups are forced to
292 also do daily Verify Jobs in order to take advantage of this useful
293 feature. This Daily Verify after Backup conduct is not always desired
294 and Verify Jobs have to be sometimes scheduled. (Not necessarily
295 scheduled in Bacula). With this feature Admins can verify Jobs once a
296 Week or less per month, selecting the Jobs they want to verify. This
297 feature is also not to difficult to implement taking in account older bug
298 reports about this feature and the selection of the Job to be verified.
300 Notes: For the verify Job, the user could select the Job to be verified
301 from a List of the latest Jobs of a client. It would also be possible to
302 verify a certain volume. All of these would naturaly apply only for
303 Jobs whose file information are still in the catalog.
306 Item 8: Improve Bacula's tape and drive usage and cleaning management
307 Date: 8 November 2005, November 11, 2005
308 Origin: Adam Thornton <athornton at sinenomine dot net>,
309 Arno Lehmann <al at its-lehmann dot de>
312 What: Make Bacula manage tape life cycle information, tape reuse
313 times and drive cleaning cycles.
315 Why: All three parts of this project are important when operating
317 We need to know which tapes need replacement, and we need to
318 make sure the drives are cleaned when necessary. While many
319 tape libraries and even autoloaders can handle all this
320 automatically, support by Bacula can be helpful for smaller
321 (older) libraries and single drives. Limiting the number of
322 times a tape is used might prevent tape errors when using
323 tapes until the drives can't read it any more. Also, checking
324 drive status during operation can prevent some failures (as I
325 [Arno] had to learn the hard way...)
327 Notes: First, Bacula could (and even does, to some limited extent)
328 record tape and drive usage. For tapes, the number of mounts,
329 the amount of data, and the time the tape has actually been
330 running could be recorded. Data fields for Read and Write
331 time and Number of mounts already exist in the catalog (I'm
332 not sure if VolBytes is the sum of all bytes ever written to
333 that volume by Bacula). This information can be important
334 when determining which media to replace. The ability to mark
335 Volumes as "used up" after a given number of write cycles
336 should also be implemented so that a tape is never actually
337 worn out. For the tape drives known to Bacula, similar
338 information is interesting to determine the device status and
339 expected life time: Time it's been Reading and Writing, number
340 of tape Loads / Unloads / Errors. This information is not yet
341 recorded as far as I [Arno] know. A new volume status would
342 be necessary for the new state, like "Used up" or "Worn out".
343 Volumes with this state could be used for restores, but not
344 for writing. These volumes should be migrated first (assuming
345 migration is implemented) and, once they are no longer needed,
346 could be moved to a Trash pool.
348 The next step would be to implement a drive cleaning setup.
349 Bacula already has knowledge about cleaning tapes. Once it
350 has some information about cleaning cycles (measured in drive
351 run time, number of tapes used, or calender days, for example)
352 it can automatically execute tape cleaning (with an
353 autochanger, obviously) or ask for operator assistance loading
356 The final step would be to implement TAPEALERT checks not only
357 when changing tapes and only sending the information to the
358 administrator, but rather checking after each tape error,
359 checking on a regular basis (for example after each tape
360 file), and also before unloading and after loading a new tape.
361 Then, depending on the drives TAPEALERT state and the known
362 drive cleaning state Bacula could automatically schedule later
363 cleaning, clean immediately, or inform the operator.
365 Implementing this would perhaps require another catalog change
366 and perhaps major changes in SD code and the DIR-SD protocol,
367 so I'd only consider this worth implementing if it would
368 actually be used or even needed by many people.
370 Implementation of these projects could happen in three distinct
371 sub-projects: Measuring Tape and Drive usage, retiring
372 volumes, and handling drive cleaning and TAPEALERTs.
375 Item 9: Allow FD to initiate a backup
376 Origin: Frank Volf (frank at deze dot org)
377 Date: 17 November 2005
380 What: Provide some means, possibly by a restricted console that
381 allows a FD to initiate a backup, and that uses the connection
382 established by the FD to the Director for the backup so that
383 a Director that is firewalled can do the backup.
384 Why: Makes backup of laptops much easier.
387 Item 10: Restore from volumes on multiple storage daemons
388 Origin: Graham Keeling (graham@equiinet.com)
392 What: The ability to restore from volumes held by multiple storage daemons
393 would be very useful.
395 Why: It is useful to be able to backup to any number of different storage
396 daemons. For example, your first storage daemon may run out of space, so you
397 switch to your second and carry on. Bacula will currently let you do this.
398 However, once you come to restore, bacula cannot cope when volumes on different
399 storage daemons are required.
401 Notes: The director knows that more than one storage daemon is needed, as
402 bconsole outputs something like the following table.
404 The job will require the following
405 Volume(s) Storage(s) SD Device(s)
406 ===========================================================================
408 backup-0001 Disk 1 Disk 1.0
409 backup-0002 Disk 2 Disk 2.0
411 However, the bootstrap file that it creates gets sent to the first storage
412 daemon only, which then stalls for a long time, 'waiting for a mount request'
413 for the volume that it doesn't have.
414 The bootstrap file contains no knowledge of the storage daemon.
415 Under the current design:
417 The director connects to the storage daemon, and gets an sd_auth_key.
418 The director then connects to the file daemon, and gives it the
419 sd_auth_key with the 'jobcmd'.
420 (restoring of files happens)
421 The director does a 'wait_for_storage_daemon_termination()'.
422 The director waits for the file daemon to indicate the end of the job.
426 The director connects to the file daemon.
427 Then, for each storage daemon in the .bsr file... {
428 The director connects to the storage daemon, and gets an sd_auth_key.
429 The director then connects to the file daemon, and gives it the
430 sd_auth_key with the 'storaddr' command.
431 (restoring of files happens)
432 The director does a 'wait_for_storage_daemon_termination()'.
433 The director waits for the file daemon to indicate the end of the
434 work on this storage.
436 The director tells the file daemon that there are no more storages to contact.
437 The director waits for the file daemon to indicate the end of the job.
439 As you can see, each restore between the file daemon and storage daemon is
440 handled in the same way that it is currently handled, using the same method
441 for authentication, except that the sd_auth_key is moved from the 'jobcmd' to
442 the 'storaddr' command - where it logically belongs.
445 Item 11: Implement Storage daemon compression
446 Date: 18 December 2006
447 Origin: Vadim A. Umanski , e-mail umanski@ext.ru
449 What: The ability to compress backup data on the SD receiving data
450 instead of doing that on client sending data.
451 Why: The need is practical. I've got some machines that can send
452 data to the network 4 or 5 times faster than compressing
453 them (I've measured that). They're using fast enough SCSI/FC
454 disk subsystems but rather slow CPUs (ex. UltraSPARC II).
455 And the backup server has got a quite fast CPUs (ex. Dual P4
456 Xeons) and quite a low load. When you have 20, 50 or 100 GB
457 of raw data - running a job 4 to 5 times faster - that
458 really matters. On the other hand, the data can be
459 compressed 50% or better - so losing twice more space for
460 disk backup is not good at all. And the network is all mine
461 (I have a dedicated management/provisioning network) and I
462 can get as high bandwidth as I need - 100Mbps, 1000Mbps...
463 That's why the server-side compression feature is needed!
467 Item 12: Reduction of communications bandwidth for a backup
468 Date: 14 October 2008
469 Origin: Robin O'Leary (Equiinet)
472 What: Using rdiff techniques, Bacula could significantly reduce
473 the network data transfer volume to do a backup.
475 Why: Faster backup across the Internet
477 Notes: This requires retaining certain data on the client during a Full
478 backup that will speed up subsequent backups.
481 Item 13: Ability to reconnect a disconnected comm line
486 What: Often jobs fail because of a communications line drop. In that
487 case, Bacula should be able to reconnect to the other daemon and
490 Why: Avoids backuping data already saved.
492 Notes: *Very* complicated from a design point of view because of authenication.
494 Item 14: Start spooling even when waiting on tape
495 Origin: Tobias Barth <tobias.barth@web-arts.com>
499 What: If a job can be spooled to disk before writing it to tape, it should
500 be spooled immediately. Currently, bacula waits until the correct
501 tape is inserted into the drive.
503 Why: It could save hours. When bacula waits on the operator who must insert
504 the correct tape (e.g. a new tape or a tape from another media
505 pool), bacula could already prepare the spooled data in the spooling
506 directory and immediately start despooling when the tape was
507 inserted by the operator.
509 2nd step: Use 2 or more spooling directories. When one directory is
510 currently despooling, the next (on different disk drives) could
511 already be spooling the next data.
513 Notes: I am using bacula 2.2.8, which has none of those features
517 Item 15: Enable/disable compression depending on storage device (disk/tape)
518 Origin: Ralf Gross ralf-lists@ralfgross.de
520 Status: Initial Request
522 What: Add a new option to the storage resource of the director. Depending
523 on this option, compression will be enabled/disabled for a device.
525 Why: If different devices (disks/tapes) are used for full/diff/incr
526 backups, software compression will be enabled for all backups
527 because of the FileSet compression option. For backup to tapes
528 wich are able to do hardware compression this is not desired.
532 http://news.gmane.org/gmane.comp.sysutils.backup.bacula.devel/cutoff=11124
533 It must be clear to the user, that the FileSet compression option
534 must still be enabled use compression for a backup job at all.
535 Thus a name for the new option in the director must be
538 Notes: KES I think the Storage definition should probably override what
539 is in the Job definition or vice-versa, but in any case, it must
543 Item 16: Include all conf files in specified directory
544 Date: 18 October 2008
545 Origin: Database, Lda. Maputo, Mozambique
546 Contact:Cameron Smith / cameron.ord@database.co.mz
549 What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every
550 time Bacula Director restarts or reloads, it will walk the given
551 directory (non-recursively) and include the contents of any files
552 therein, as though they were appended to bacula-dir.conf
554 Why: Permits simplified and safer configuration for larger installations with
555 many client PCs. Currently, through judicious use of JobDefs and
556 similar directives, it is possible to reduce the client-specific part of
557 a configuration to a minimum. The client-specific directives can be
558 prepared according to a standard template and dropped into a known
559 directory. However it is still necessary to add a line to the "master"
560 (bacula-dir.conf) referencing each new file. This exposes the master to
561 unnecessary risk of accidental mistakes and makes automation of adding
562 new client-confs, more difficult (it is easier to automate dropping a
563 file into a dir, than rewriting an existing file). Ken has previously
564 made a convincing argument for NOT including Bacula's core configuration
565 in an RDBMS, but I believe that the present request is a reasonable
566 extension to the current "flat-file-based" configuration philosophy.
568 Notes: There is NO need for any special syntax to these files. They should
569 contain standard directives which are simply "inlined" to the parent
570 file as already happens when you explicitly reference an external file.
572 Notes: (kes) this can already be done with scripting
573 From: John Jorgensen <jorgnsn@lcd.uregina.ca>
574 The bacula-dir.conf at our site contains these lines:
577 # Include subfiles associated with configuration of clients.
578 # They define the bulk of the Clients, Jobs, and FileSets.
580 @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'"
582 and when we get a new client, we just put its configuration into
583 a new file called something like:
585 /etc/bacula/clientdefs/clientname.conf
588 Item 17: Multiple threads in file daemon for the same job
589 Date: 27 November 2005
590 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
593 What: I want the file daemon to start multiple threads for a backup
594 job so the fastest possible backup can be made.
596 The file daemon could parse the FileSet information and start
597 one thread for each File entry located on a separate
600 A confiuration option in the job section should be used to
601 enable or disable this feature. The confgutration option could
602 specify the maximum number of threads in the file daemon.
604 If the theads could spool the data to separate spool files
605 the restore process will not be much slower.
607 Why: Multiple concurrent backups of a large fileserver with many
608 disks and controllers will be much faster.
611 Item 18: Possibilty to schedule Jobs on last Friday of the month
612 Origin: Carsten Menke <bootsy52 at gmx dot net>
616 What: Currently if you want to run your monthly Backups on the last
617 Friday of each month this is only possible with workarounds (e.g
618 scripting) (As some months got 4 Fridays and some got 5 Fridays)
619 The same is true if you plan to run your yearly Backups on the
620 last Friday of the year. It would be nice to have the ability to
621 use the builtin scheduler for this.
623 Why: In many companies the last working day of the week is Friday (or
624 Saturday), so to get the most data of the month onto the monthly
625 tape, the employees are advised to insert the tape for the
626 monthly backups on the last friday of the month.
628 Notes: To give this a complete functionality it would be nice if the
629 "first" and "last" Keywords could be implemented in the
630 scheduler, so it is also possible to run monthy backups at the
631 first friday of the month and many things more. So if the syntax
632 would expand to this {first|last} {Month|Week|Day|Mo-Fri} of the
633 {Year|Month|Week} you would be able to run really flexible jobs.
635 To got a certain Job run on the last Friday of the Month for example one could
638 Run = pool=Monthly last Fri of the Month at 23:50
642 Run = pool=Yearly last Fri of the Year at 23:50
644 ## Certain Jobs the last Week of a Month
646 Run = pool=LastWeek last Week of the Month at 23:50
648 ## Monthly Backup on the last day of the month
650 Run = pool=Monthly last Day of the Month at 23:50
653 Item 19: Include timestamp of job launch in "stat clients" output
654 Origin: Mark Bergman <mark.bergman@uphs.upenn.edu>
655 Date: Tue Aug 22 17:13:39 EDT 2006
658 What: The "stat clients" command doesn't include any detail on when
659 the active backup jobs were launched.
661 Why: Including the timestamp would make it much easier to decide whether
662 a job is running properly.
664 Notes: It may be helpful to have the output from "stat clients" formatted
665 more like that from "stat dir" (and other commands), in a column
666 format. The per-client information that's currently shown (level,
667 client name, JobId, Volume, pool, device, Files, etc.) is good, but
668 somewhat hard to parse (both programmatically and visually),
669 particularly when there are many active clients.
672 Item 20: Cause daemons to use a specific IP address to source communications
673 Origin: Bill Moran <wmoran@collaborativefusion.com>
676 What: Cause Bacula daemons (dir, fd, sd) to always use the ip address
677 specified in the [DIR|DF|SD]Addr directive as the source IP
678 for initiating communication.
679 Why: On complex networks, as well as extremely secure networks, it's
680 not unusual to have multiple possible routes through the network.
681 Often, each of these routes is secured by different policies
682 (effectively, firewalls allow or deny different traffic depending
683 on the source address)
684 Unfortunately, it can sometimes be difficult or impossible to
685 represent this in a system routing table, as the result is
686 excessive subnetting that quickly exhausts available IP space.
687 The best available workaround is to provide multiple IPs to
688 a single machine that are all on the same subnet. In order
689 for this to work properly, applications must support the ability
690 to bind outgoing connections to a specified address, otherwise
691 the operating system will always choose the first IP that
692 matches the required route.
693 Notes: Many other programs support this. For example, the following
694 can be configured in BIND:
695 query-source address 10.0.0.1;
696 transfer-source 10.0.0.2;
697 Which means queries from this server will always come from
698 10.0.0.1 and zone transfers will always originate from
702 Item 21: Message mailing based on backup types
703 Origin: Evan Kaufman <evan.kaufman@gmail.com>
704 Date: January 6, 2006
707 What: In the "Messages" resource definitions, allowing messages
708 to be mailed based on the type (backup, restore, etc.) and level
709 (full, differential, etc) of job that created the originating
712 Why: It would, for example, allow someone's boss to be emailed
713 automatically only when a Full Backup job runs, so he can
714 retrieve the tapes for offsite storage, even if the IT dept.
715 doesn't (or can't) explicitly notify him. At the same time, his
716 mailbox wouldnt be filled by notifications of Verifies, Restores,
717 or Incremental/Differential Backups (which would likely be kept
720 Notes: One way this could be done is through additional message types, for example:
723 # email the boss only on full system backups
724 Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
726 # email us only when something breaks
727 MailOnError = itdept@mycompany.com = all
730 Notes: Kern: This should be rather trivial to implement.
733 Item 22: Ability to import/export Bacula database entities
738 What: Create a Bacula ASCII SQL database independent format that permits
739 importing and exporting database catalog Job entities.
741 Why: For achival, database clustering, tranfer to other databases
744 Notes: Job selection should be by Job, time, Volume, Client, Pool and possibly
748 Item 23: "Maximum Concurrent Jobs" for drives when used with changer device
749 Origin: Ralf Gross ralf-lists <at> ralfgross.de
751 Status: Initial Request
753 What: respect the "Maximum Concurrent Jobs" directive in the _drives_
754 Storage section in addition to the changer section
756 Why: I have a 3 drive changer where I want to be able to let 3 concurrent
757 jobs run in parallel. But only one job per drive at the same time.
758 Right now I don't see how I could limit the number of concurrent jobs
759 per drive in this situation.
761 Notes: Using different priorities for these jobs lead to problems that other
762 jobs are blocked. On the user list I got the advice to use the "Prefer Mounted
763 Volumes" directive, but Kern advised against using "Prefer Mounted
764 Volumes" in an other thread:
765 http://article.gmane.org/gmane.comp.sysutils.backup.bacula.devel/11876/
767 In addition I'm not sure if this would be the same as respecting the
768 drive's "Maximum Concurrent Jobs" setting.
780 Maximum Concurrent Jobs = 3
784 Name = Neo4100-LTO4-D1
788 Device = ULTRIUM-TD4-D1
790 Maximum Concurrent Jobs = 1
795 The "Maximum Concurrent Jobs = 1" directive in the drive's section is ignored.
798 Item 24: Implementation of running Job speed limit.
799 Origin: Alex F, alexxzell at yahoo dot com
800 Date: 29 January 2009
802 What: I noticed the need for an integrated bandwidth limiter for
803 running jobs. It would be very useful just to specify another
804 field in bacula-dir.conf, like speed = how much speed you wish
805 for that specific job to run at
807 Why: Because of a couple of reasons. First, it's very hard to implement a
808 traffic shaping utility and also make it reliable. Second, it is very
809 uncomfortable to have to implement these apps to, let's say 50 clients
810 (including desktops, servers). This would also be unreliable because you
811 have to make sure that the apps are properly working when needed; users
812 could also disable them (accidentally or not). It would be very useful
813 to provide Bacula this ability. All information would be centralized,
814 you would not have to go to 50 different clients in 10 different
815 locations for configuration; eliminating 3rd party additions help in
816 establishing efficiency. Would also avoid bandwidth congestion,
817 especially where there is little available.
820 Item 25: Add an override in Schedule for Pools based on backup types
822 Origin: Chad Slater <chad.slater@clickfox.com>
825 What: Adding a FullStorage=BigTapeLibrary in the Schedule resource
826 would help those of us who use different storage devices for different
827 backup levels cope with the "auto-upgrade" of a backup.
829 Why: Assume I add several new devices to be backed up, i.e. several
830 hosts with 1TB RAID. To avoid tape switching hassles, incrementals are
831 stored in a disk set on a 2TB RAID. If you add these devices in the
832 middle of the month, the incrementals are upgraded to "full" backups,
833 but they try to use the same storage device as requested in the
834 incremental job, filling up the RAID holding the differentials. If we
835 could override the Storage parameter for full and/or differential
836 backups, then the Full job would use the proper Storage device, which
837 has more capacity (i.e. a 8TB tape library.
840 Item 26: Automatic promotion of backup levels based on backup size
841 Date: 19 January 2006
842 Origin: Adam Thornton <athornton@sinenomine.net>
845 What: Other backup programs have a feature whereby it estimates the space
846 that a differential, incremental, and full backup would take. If
847 the difference in space required between the scheduled level and the
848 next level up is beneath some user-defined critical threshold, the
849 backup level is bumped to the next type. Doing this minimizes the
850 number of volumes necessary during a restore, with a fairly minimal
851 cost in backup media space.
853 Why: I know at least one (quite sophisticated and smart) user for whom the
854 absence of this feature is a deal-breaker in terms of using Bacula;
855 if we had it it would eliminate the one cool thing other backup
856 programs can do and we can't (at least, the one cool thing I know
860 Item 27: Allow inclusion/exclusion of files in a fileset by creation/mod times
861 Origin: Evan Kaufman <evan.kaufman@gmail.com>
862 Date: January 11, 2006
865 What: In the vein of the Wild and Regex directives in a Fileset's
866 Options, it would be helpful to allow a user to include or exclude
867 files and directories by creation or modification times.
869 You could factor the Exclude=yes|no option in much the same way it
870 affects the Wild and Regex directives. For example, you could exclude
871 all files modified before a certain date:
875 Modified Before = ####
878 Or you could exclude all files created/modified since a certain date:
882 Created Modified Since = ####
885 The format of the time/date could be done several ways, say the number
886 of seconds since the epoch:
887 1137008553 = Jan 11 2006, 1:42:33PM # result of `date +%s`
889 Or a human readable date in a cryptic form:
890 20060111134233 = Jan 11 2006, 1:42:33PM # YYYYMMDDhhmmss
892 Why: I imagine a feature like this could have many uses. It would
893 allow a user to do a full backup while excluding the base operating
894 system files, so if I installed a Linux snapshot from a CD yesterday,
895 I'll *exclude* all files modified *before* today. If I need to
896 recover the system, I use the CD I already have, plus the tape backup.
897 Or if, say, a Windows client is hit by a particularly corrosive
898 virus, and I need to *exclude* any files created/modified *since* the
901 Notes: Of course, this feature would work in concert with other
902 in/exclude rules, and wouldnt override them (or each other).
904 Notes: The directives I'd imagine would be along the lines of
905 "[Created] [Modified] [Before|Since] = <date>".
906 So one could compare against 'ctime' and/or 'mtime', but ONLY 'before'
910 Item 28: Archival (removal) of User Files to Tape
912 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
915 What: The ability to archive data to storage based on certain parameters
916 such as age, size, or location. Once the data has been written to
917 storage and logged it is then pruned from the originating
918 filesystem. Note! We are talking about user's files and not
921 Why: This would allow fully automatic storage management which becomes
922 useful for large datastores. It would also allow for auto-staging
923 from one media type to another.
925 Example 1) Medical imaging needs to store large amounts of data.
926 They decide to keep data on their servers for 6 months and then put
927 it away for long term storage. The server then finds all files
928 older than 6 months writes them to tape. The files are then removed
931 Example 2) All data that hasn't been accessed in 2 months could be
932 moved from high-cost, fibre-channel disk storage to a low-cost
933 large-capacity SATA disk storage pool which doesn't have as quick of
934 access time. Then after another 6 months (or possibly as one
935 storage pool gets full) data is migrated to Tape.
938 Item 29: An option to operate on all pools with update vol parameters
939 Origin: Dmitriy Pinchukov <absh@bossdev.kiev.ua>
941 Status: Patch made by Nigel Stepp
943 What: When I do update -> Volume parameters -> All Volumes
944 from Pool, then I have to select pools one by one. I'd like
945 console to have an option like "0: All Pools" in the list of
948 Why: I have many pools and therefore unhappy with manually
949 updating each of them using update -> Volume parameters -> All
950 Volumes from Pool -> pool #.
953 Item 30: Automatic disabling of devices
955 Origin: Peter Eriksson <peter at ifm.liu dot se>
958 What: After a configurable amount of fatal errors with a tape drive
959 Bacula should automatically disable further use of a certain
960 tape drive. There should also be "disable"/"enable" commands in
963 Why: On a multi-drive jukebox there is a possibility of tape drives
964 going bad during large backups (needing a cleaning tape run,
965 tapes getting stuck). It would be advantageous if Bacula would
966 automatically disable further use of a problematic tape drive
967 after a configurable amount of errors has occurred.
969 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
970 where tapes occasionally get stuck inside the drive. Bacula will
971 notice that the "mtx-changer" command will fail and then fail
972 any backup jobs trying to use that drive. However, it will still
973 keep on trying to run new jobs using that drive and fail -
974 forever, and thus failing lots and lots of jobs... Since we have
975 many drives Bacula could have just automatically disabled
976 further use of that drive and used one of the other ones
980 Item 31: List InChanger flag when doing restore.
981 Origin: Jesper Krogh<jesper@krogh.cc>
985 What: When doing a restore the restore selection dialog ends by telling stuff
987 The job will require the following
988 Volume(s) Storage(s) SD Device(s)
989 ===========================================================================
1000 When having an autochanger, it would be really nice with an inChanger
1001 column so the operator knew if this restore job would stop waiting for
1002 operator intervention. This is done just by selecting the inChanger flag
1003 from the catalog and printing it in a seperate column.
1006 Why: This would help getting large restores through minimizing the
1007 time spent waiting for operator to drop by and change tapes in the library.
1009 Notes: [Kern] I think it would also be good to have the Slot as well,
1010 or some indication that Bacula thinks the volume is in the autochanger
1011 because it depends on both the InChanger flag and the Slot being
1015 Item 32: Ability to defer Batch Insert to a later time
1020 What: Instead of doing a Job Batch Insert at the end of the Job
1021 which might create resource contention with lots of Job,
1022 defer the insert to a later time.
1024 Why: Permits to focus on getting the data on the Volume and
1025 putting the metadata into the Catalog outside the backup
1028 Notes: Will use the proposed Bacula ASCII database import/export
1029 format (i.e. dependent on the import/export entities project).
1032 Item 33: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
1033 Origin: Bastian Friedrich <bastian.friedrich@collax.com>
1037 What: SD has a "Maximum Volume Size" statement, which is deprecated and
1038 superseded by the Pool resource statement "Maximum Volume Bytes".
1039 It would be good if either statement could be used in Storage
1042 Why: Pools do not have to be restricted to a single storage type/device;
1043 thus, it may be impossible to define Maximum Volume Bytes in the
1044 Pool resource. The old MaxVolSize statement is deprecated, as it
1045 is SD side only. I am using the same pool for different devices.
1047 Notes: State of idea currently unknown. Storage resources in the dir
1048 config currently translate to very slim catalog entries; these
1049 entries would require extensions to implement what is described
1050 here. Quite possibly, numerous other statements that are currently
1051 available in Pool resources could be used in Storage resources too
1055 Item 34: Enable persistent naming/number of SQL queries
1057 Origin: Mark Bergman
1061 Change the parsing of the query.sql file and the query command so that
1062 queries are named/numbered by a fixed value, not their order in the
1067 One of the real strengths of bacula is the ability to query the
1068 database, and the fact that complex queries can be saved and
1069 referenced from a file is very powerful. However, the choice
1070 of query (both for interactive use, and by scripting input
1071 to the bconsole command) is completely dependent on the order
1072 within the query.sql file. The descriptve labels are helpful for
1073 interactive use, but users become used to calling a particular
1074 query "by number", or may use scripts to execute queries. This
1075 presents a problem if the number or order of queries in the file
1078 If the query.sql file used the numeric tags as a real value (rather
1079 than a comment), then users could have a higher confidence that they
1080 are executing the intended query, that their local changes wouldn't
1081 conflict with future bacula upgrades.
1083 For scripting, it's very important that the intended query is
1084 what's actually executed. The current method of parsing the
1085 query.sql file discourages scripting because the addition or
1086 deletion of queries within the file will require corresponding
1087 changes to scripts. It may not be obvious to users that deleting
1088 query "17" in the query.sql file will require changing all
1089 references to higher numbered queries. Similarly, when new
1090 bacula distributions change the number of "official" queries,
1091 user-developed queries cannot simply be appended to the file
1092 without also changing any references to those queries in scripts
1093 or procedural documentation, etc.
1095 In addition, using fixed numbers for queries would encourage more
1096 user-initiated development of queries, by supporting conventions
1099 queries numbered 1-50 are supported/developed/distributed by
1100 with official bacula releases
1102 queries numbered 100-200 are community contributed, and are
1103 related to media management
1105 queries numbered 201-300 are community contributed, and are
1106 related to checksums, finding duplicated files across
1107 different backups, etc.
1109 queries numbered 301-400 are community contributed, and are
1110 related to backup statistics (average file size, size per
1111 client per backup level, time for all clients by backup level,
1112 storage capacity by media type, etc.)
1114 queries numbered 500-999 are locally created
1117 Alternatively, queries could be called by keyword (tag), rather
1121 Item 35: Port bat to Win32
1126 What: Make bat run on Win32/64.
1128 Why: To have GUI on Windows
1133 Item 36: Bacula Dir, FD and SD to support proxies
1134 Origin: Karl Grindley @ MIT Lincoln Laboratory <kgrindley at ll dot mit dot edu>
1138 What: Support alternate methods for nailing up a TCP session such
1139 as SOCKS5, SOCKS4 and HTTP (CONNECT) proxies. Such a feature
1140 would allow tunneling of bacula traffic in and out of proxied
1143 Why: Currently, bacula is architected to only function on a flat network, with
1144 no barriers or limitations. Due to the large configuration states of
1145 any network and the infinite configuration where file daemons and
1146 storage daemons may sit in relation to one another, bacula often is
1147 not usable on a network where filtered or air-gaped networks exist.
1148 While often solutions such as ACL modifications to firewalls or port
1149 redirection via SNAT or DNAT will solve the issue, often however,
1150 these solutions are not adequate or not allowed by hard policy.
1152 In an air-gapped network with only a highly locked down proxy services
1153 are provided (SOCKS4/5 and/or HTTP and/or SSH outbound) ACLs or
1154 iptable rules will not work.
1156 Notes: Director resource tunneling: This configuration option to utilize a
1157 proxy to connect to a client should be specified in the client
1158 resource Client resource tunneling: should be configured in the client
1159 resource in the director config file? Or configured on the bacula-fd
1160 configuration file on the fd host itself? If the ladder, this would
1161 allow only certain clients to use a proxy, where others do not when
1162 establishing the TCP connection to the storage server.
1164 Also worth noting, there are other 3rd party, light weight apps that
1165 could be utilized to bootstrap this. Instead of sockifing bacula
1166 itself, use an external program to broker proxy authentication, and
1167 connection to the remote host. OpenSSH does this by using the
1168 "ProxyCommand" syntax in the client configuration and uses stdin and
1169 stdout to the command. Connect.c is a very popular one.
1170 (http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html).
1171 One could also possibly use stunnel, netcat, etc.
1174 Item 37: Add Minumum Spool Size directive
1176 Origin: Frank Sweetser <fs@wpi.edu>
1178 What: Add a new SD directive, "minimum spool size" (or similar). This
1179 directive would specify a minimum level of free space available for
1180 spooling. If the unused spool space is less than this level, any
1181 new spooling requests would be blocked as if the "maximum spool
1182 size" threshold had bee reached. Already spooling jobs would be
1183 unaffected by this directive.
1185 Why: I've been bitten by this scenario a couple of times:
1187 Assume a maximum spool size of 100M. Two concurrent jobs, A and B,
1188 are both running. Due to timing quirks and previously running jobs,
1189 job A has used 99.9M of space in the spool directory. While A is
1190 busy despooling to disk, B is happily using the remaining 0.1M of
1191 spool space. This ends up in a spool/despool sequence every 0.1M of
1192 data. In addition to fragmenting the data on the volume far more
1193 than was necessary, in larger data sets (ie, tens or hundreds of
1194 gigabytes) it can easily produce multi-megabyte report emails!
1197 Item 38: Backup and Restore of Windows Encrypted Files using Win raw encryption
1198 Origin: Michael Mohr, SAG Mohr.External@infineon.com
1199 Date: 22 February 2008
1200 Origin: Alex Ehrlich (Alex.Ehrlich-at-mail.ee)
1201 Date: 05 August 2008
1204 What: Make it possible to backup and restore Encypted Files from and to
1205 Windows systems without the need to decrypt it by using the raw
1206 encryption functions API (see:
1207 http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
1208 that is provided for that reason by Microsoft.
1209 If a file ist encrypted could be examined by evaluating the
1210 FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes
1212 For each file backed up or restored by FD on Windows, check if
1213 the file is encrypted; if so then use OpenEncryptedFileRaw,
1214 ReadEncryptedFileRaw, WriteEncryptedFileRaw,
1215 CloseEncryptedFileRaw instead of BackupRead and BackupWrite
1218 Why: Without the usage of this interface the fd-daemon running
1219 under the system account can't read encypted Files because
1220 the key needed for the decrytion is missed by them. As a result
1221 actually encrypted files are not backed up
1222 by bacula and also no error is shown while missing these files.
1224 Notes: Using xxxEncryptedFileRaw API would allow to backup and
1225 restore EFS-encrypted files without decrypting their data.
1226 Note that such files cannot be restored "portably" (at least,
1227 easily) but they would be restoreable to a different (or
1228 reinstalled) Win32 machine; the restore would require setup
1229 of a EFS recovery agent in advance, of course, and this shall
1230 be clearly reflected in the documentation, but this is the
1231 normal Windows SysAdmin's business.
1232 When "portable" backup is requested the EFS-encrypted files
1233 shall be clearly reported as errors.
1234 See MSDN on the "Backup and Restore of Encrypted Files" topic:
1235 http://msdn.microsoft.com/en-us/library/aa363783.aspx
1236 Maybe the EFS support requires a new flag in the database for
1238 Unfortunately, the implementation is not as straightforward as
1239 1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
1240 requiring some FD code rewrite to work with
1241 encrypted-file-related callback functions.
1244 Item 39: Implement an interface between Bacula and Storage clould like Amazon's S3.
1245 Date: 25 August 2008
1246 Origin: Soren Hansen <soren@ubuntu.com>
1247 Status: Not started.
1248 What: Enable the storage daemon to store backup data on Amazon's
1251 Why: Amazon's S3 is a cheap way to store data off-site.
1253 Notes: If we configure the Pool to put only one job per volume (they don't
1254 support append operation), and the volume size isn't to big (100MB?),
1255 it should be easy to adapt the disk-changer script to add get/put
1256 procedure with curl. So, the data would be safetly copied during the
1259 Cloud should be only used with Copy jobs, users should always have
1260 a copy of their data on their site.
1262 We should also think to have our own cache, trying always to have
1263 cloud volume on the local disk. (I don't know if users want to store
1264 100GB on cloud, so it shouldn't be a disk size problem). For example,
1265 if bacula want to recycle a volume, it will start by downloading the
1266 file to truncate it few seconds later, if we can avoid that...
1268 Item 40: Convert Bacula existing tray monitor on Windows to a stand alone program
1273 What: Separate Win32 tray monitor to be a separate program.
1275 Why: Vista does not allow SYSTEM services to interact with the
1276 desktop, so the current tray monitor does not work on Vista
1279 Notes: Requires communicating with the FD via the network (simulate
1280 a console connection).
1284 ========= End items voted on May 2009 ==================
1286 ========= New items after last vote ====================
1288 Item 1: Relabel disk volume after recycling
1289 Origin: Pasi Kärkkäinen <pasik@iki.fi>
1291 Status: Not implemented yet, no code written.
1293 What: The ability to relabel the disk volume (and thus rename the file on the disk)
1294 after it has been recycled. Useful when you have a single job per disk volume,
1295 and you use a custom Label format, for example:
1296 Label Format = "${Client}-${Level}-${NumVols:p/4/0/r}-${Year}_${Month}_${Day}-${Hour}_${Minute}"
1298 Why: Disk volumes in Bacula get the label/filename when they are used for the first time.
1299 If you use recycling and custom label format like above, the disk
1300 volume name doesn't match the contents after it has been recycled.
1301 This feature makes it possible to keep the label/filename in sync
1302 with the content and thus makes it easy to check/monitor the backups
1303 from the shell and/or normal file management tools, because the filenames
1304 of the disk volumes match the content.
1306 Notes: The configuration option could be "Relabel after Recycling = Yes".
1309 ========= Add new items above this line =================
1312 ============= Empty Feature Request form ===========
1313 Item n: One line summary ...
1314 Date: Date submitted
1315 Origin: Name and email of originator.
1318 What: More detailed explanation ...
1320 Why: Why it is important ...
1322 Notes: Additional notes or features (omit if not used)
1323 ============== End Feature Request form ==============
1326 ========== Items put on hold by Kern ============================