3 Bacula Projects Roadmap
4 Status updated 25 February 2010
9 Item 1: Ability to restart failed jobs
10 Item 3: Scheduling syntax that permits more flexibility and options
11 Item 4: Data encryption on storage daemon
12 Item 7: Add ability to Verify any specified Job.
13 Item 8: Improve Bacula's tape and drive usage and cleaning management
14 Item 9: Allow FD to initiate a backup
15 Item 11: Implement Storage daemon compression
16 Item 12: Reduction of communications bandwidth for a backup
17 Item 13: Ability to reconnect a disconnected comm line
18 Item 14: Start spooling even when waiting on tape
19 Item 16: Include all conf files in specified directory
20 Item 17: Multiple threads in file daemon for the same job
21 Item 18: Possibilty to schedule Jobs on last Friday of the month
22 Item 19: Include timestamp of job launch in "stat clients" output
23 Item 21: Message mailing based on backup types
24 Item 22: Ability to import/export Bacula database entities
25 Item 24: Implementation of running Job speed limit.
26 Item 25: Add an override in Schedule for Pools based on backup types
27 Item 26: Automatic promotion of backup levels based on backup size
28 Item 27: Allow inclusion/exclusion of files in a fileset by creation/mod times
29 Item 28: Archival (removal) of User Files to Tape
30 Item 29: An option to operate on all pools with update vol parameters
31 Item 30: Automatic disabling of devices
32 Item 32: Ability to defer Batch Insert to a later time
33 Item 33: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
34 Item 34: Enable persistent naming/number of SQL queries
35 Item 36: Bacula Dir, FD and SD to support proxies
36 Item 37: Add Minumum Spool Size directive
37 Item 38: Backup and Restore of Windows Encrypted Files using Win raw encryption
38 Item 39: Implement an interface between Bacula and Amazon's S3.
39 Item 40: Convert Bacula existing tray monitor on Windows to a stand alone program
43 Item 1: Ability to restart failed jobs
48 What: Often jobs fail because of a communications line drop or max run time,
49 cancel, or some other non-critical problem. Currrently any data
50 saved is lost. This implementation should modify the Storage daemon
51 so that it saves all the files that it knows are completely backed
54 The jobs should then be marked as incomplete and a subsequent
55 Incremental Accurate backup will then take into account all the
58 Why: Avoids backuping data already saved.
60 Notes: Requires Accurate to restart correctly. Must completed have a minimum
61 volume of data or files stored on Volume before enabling.
64 Item 3: Scheduling syntax that permits more flexibility and options
65 Date: 15 December 2006
66 Origin: Gregory Brauer (greg at wildbrain dot com) and
67 Florian Schnabel <florian.schnabel at docufy dot de>
70 What: Currently, Bacula only understands how to deal with weeks of the
71 month or weeks of the year in schedules. This makes it impossible
72 to do a true weekly rotation of tapes. There will always be a
73 discontinuity that will require disruptive manual intervention at
74 least monthly or yearly because week boundaries never align with
75 month or year boundaries.
77 A solution would be to add a new syntax that defines (at least)
78 a start timestamp, and repetition period.
80 An easy option to skip a certain job on a certain date.
83 Why: Rotated backups done at weekly intervals are useful, and Bacula
84 cannot currently do them without extensive hacking.
86 You could then easily skip tape backups on holidays. Especially
87 if you got no autochanger and can only fit one backup on a tape
88 that would be really handy, other jobs could proceed normally
89 and you won't get errors that way.
92 Notes: Here is an example syntax showing a 3-week rotation where full
93 Backups would be performed every week on Saturday, and an
94 incremental would be performed every week on Tuesday. Each
95 set of tapes could be removed from the loader for the following
96 two cycles before coming back and being reused on the third
97 week. Since the execution times are determined by intervals
98 from a given point in time, there will never be any issues with
99 having to adjust to any sort of arbitrary time boundary. In
100 the example provided, I even define the starting schedule
101 as crossing both a year and a month boundary, but the run times
102 would be based on the "Repeat" value and would therefore happen
107 Name = "Week 1 Rotation"
108 #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
112 Start = 2006-12-30 01:00
116 #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
120 Start = 2007-01-02 01:00
127 Name = "Week 2 Rotation"
128 #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
132 Start = 2007-01-06 01:00
136 #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
140 Start = 2007-01-09 01:00
147 Name = "Week 3 Rotation"
148 #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
152 Start = 2007-01-13 01:00
156 #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
160 Start = 2007-01-16 01:00
166 Notes: Kern: I have merged the previously separate project of skipping
167 jobs (via Schedule syntax) into this.
170 Item 4: Data encryption on storage daemon
171 Origin: Tobias Barth <tobias.barth at web-arts.com>
172 Date: 04 February 2009
175 What: The storage demon should be able to do the data encryption that can
176 currently be done by the file daemon.
178 Why: This would have 2 advantages:
179 1) one could encrypt the data of unencrypted tapes by doing a
181 2) the storage daemon would be the only machine that would have
182 to keep the encryption keys.
185 As an addendum to the feature request, here are some crypto
186 implementation details I wrote up regarding SD-encryption back in Jan
188 http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html
191 Item 7: Add ability to Verify any specified Job.
192 Date: 17 January 2008
193 Origin: portrix.net Hamburg, Germany.
194 Contact: Christian Sabelmann
195 Status: 70% of the required Code is part of the Verify function since v. 2.x
198 The ability to tell Bacula which Job should verify instead of
199 automatically verify just the last one.
202 It is sad that such a powerfull feature like Verify Jobs
203 (VolumeToCatalog) is restricted to be used only with the last backup Job
204 of a client. Actual users who have to do daily Backups are forced to
205 also do daily Verify Jobs in order to take advantage of this useful
206 feature. This Daily Verify after Backup conduct is not always desired
207 and Verify Jobs have to be sometimes scheduled. (Not necessarily
208 scheduled in Bacula). With this feature Admins can verify Jobs once a
209 Week or less per month, selecting the Jobs they want to verify. This
210 feature is also not to difficult to implement taking in account older bug
211 reports about this feature and the selection of the Job to be verified.
213 Notes: For the verify Job, the user could select the Job to be verified
214 from a List of the latest Jobs of a client. It would also be possible to
215 verify a certain volume. All of these would naturaly apply only for
216 Jobs whose file information are still in the catalog.
219 Item 8: Improve Bacula's tape and drive usage and cleaning management
220 Date: 8 November 2005, November 11, 2005
221 Origin: Adam Thornton <athornton at sinenomine dot net>,
222 Arno Lehmann <al at its-lehmann dot de>
225 What: Make Bacula manage tape life cycle information, tape reuse
226 times and drive cleaning cycles.
228 Why: All three parts of this project are important when operating
230 We need to know which tapes need replacement, and we need to
231 make sure the drives are cleaned when necessary. While many
232 tape libraries and even autoloaders can handle all this
233 automatically, support by Bacula can be helpful for smaller
234 (older) libraries and single drives. Limiting the number of
235 times a tape is used might prevent tape errors when using
236 tapes until the drives can't read it any more. Also, checking
237 drive status during operation can prevent some failures (as I
238 [Arno] had to learn the hard way...)
240 Notes: First, Bacula could (and even does, to some limited extent)
241 record tape and drive usage. For tapes, the number of mounts,
242 the amount of data, and the time the tape has actually been
243 running could be recorded. Data fields for Read and Write
244 time and Number of mounts already exist in the catalog (I'm
245 not sure if VolBytes is the sum of all bytes ever written to
246 that volume by Bacula). This information can be important
247 when determining which media to replace. The ability to mark
248 Volumes as "used up" after a given number of write cycles
249 should also be implemented so that a tape is never actually
250 worn out. For the tape drives known to Bacula, similar
251 information is interesting to determine the device status and
252 expected life time: Time it's been Reading and Writing, number
253 of tape Loads / Unloads / Errors. This information is not yet
254 recorded as far as I [Arno] know. A new volume status would
255 be necessary for the new state, like "Used up" or "Worn out".
256 Volumes with this state could be used for restores, but not
257 for writing. These volumes should be migrated first (assuming
258 migration is implemented) and, once they are no longer needed,
259 could be moved to a Trash pool.
261 The next step would be to implement a drive cleaning setup.
262 Bacula already has knowledge about cleaning tapes. Once it
263 has some information about cleaning cycles (measured in drive
264 run time, number of tapes used, or calender days, for example)
265 it can automatically execute tape cleaning (with an
266 autochanger, obviously) or ask for operator assistance loading
269 The final step would be to implement TAPEALERT checks not only
270 when changing tapes and only sending the information to the
271 administrator, but rather checking after each tape error,
272 checking on a regular basis (for example after each tape
273 file), and also before unloading and after loading a new tape.
274 Then, depending on the drives TAPEALERT state and the known
275 drive cleaning state Bacula could automatically schedule later
276 cleaning, clean immediately, or inform the operator.
278 Implementing this would perhaps require another catalog change
279 and perhaps major changes in SD code and the DIR-SD protocol,
280 so I'd only consider this worth implementing if it would
281 actually be used or even needed by many people.
283 Implementation of these projects could happen in three distinct
284 sub-projects: Measuring Tape and Drive usage, retiring
285 volumes, and handling drive cleaning and TAPEALERTs.
288 Item 9: Allow FD to initiate a backup
289 Origin: Frank Volf (frank at deze dot org)
290 Date: 17 November 2005
293 What: Provide some means, possibly by a restricted console that
294 allows a FD to initiate a backup, and that uses the connection
295 established by the FD to the Director for the backup so that
296 a Director that is firewalled can do the backup.
297 Why: Makes backup of laptops much easier.
300 Item 11: Implement Storage daemon compression
301 Date: 18 December 2006
302 Origin: Vadim A. Umanski , e-mail umanski@ext.ru
304 What: The ability to compress backup data on the SD receiving data
305 instead of doing that on client sending data.
306 Why: The need is practical. I've got some machines that can send
307 data to the network 4 or 5 times faster than compressing
308 them (I've measured that). They're using fast enough SCSI/FC
309 disk subsystems but rather slow CPUs (ex. UltraSPARC II).
310 And the backup server has got a quite fast CPUs (ex. Dual P4
311 Xeons) and quite a low load. When you have 20, 50 or 100 GB
312 of raw data - running a job 4 to 5 times faster - that
313 really matters. On the other hand, the data can be
314 compressed 50% or better - so losing twice more space for
315 disk backup is not good at all. And the network is all mine
316 (I have a dedicated management/provisioning network) and I
317 can get as high bandwidth as I need - 100Mbps, 1000Mbps...
318 That's why the server-side compression feature is needed!
322 Item 12: Reduction of communications bandwidth for a backup
323 Date: 14 October 2008
324 Origin: Robin O'Leary (Equiinet)
327 What: Using rdiff techniques, Bacula could significantly reduce
328 the network data transfer volume to do a backup.
330 Why: Faster backup across the Internet
332 Notes: This requires retaining certain data on the client during a Full
333 backup that will speed up subsequent backups.
336 Item 13: Ability to reconnect a disconnected comm line
341 What: Often jobs fail because of a communications line drop. In that
342 case, Bacula should be able to reconnect to the other daemon and
345 Why: Avoids backuping data already saved.
347 Notes: *Very* complicated from a design point of view because of authenication.
349 Item 14: Start spooling even when waiting on tape
350 Origin: Tobias Barth <tobias.barth@web-arts.com>
354 What: If a job can be spooled to disk before writing it to tape, it should
355 be spooled immediately. Currently, bacula waits until the correct
356 tape is inserted into the drive.
358 Why: It could save hours. When bacula waits on the operator who must insert
359 the correct tape (e.g. a new tape or a tape from another media
360 pool), bacula could already prepare the spooled data in the spooling
361 directory and immediately start despooling when the tape was
362 inserted by the operator.
364 2nd step: Use 2 or more spooling directories. When one directory is
365 currently despooling, the next (on different disk drives) could
366 already be spooling the next data.
368 Notes: I am using bacula 2.2.8, which has none of those features
372 Item 16: Include all conf files in specified directory
373 Date: 18 October 2008
374 Origin: Database, Lda. Maputo, Mozambique
375 Contact:Cameron Smith / cameron.ord@database.co.mz
378 What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every
379 time Bacula Director restarts or reloads, it will walk the given
380 directory (non-recursively) and include the contents of any files
381 therein, as though they were appended to bacula-dir.conf
383 Why: Permits simplified and safer configuration for larger installations with
384 many client PCs. Currently, through judicious use of JobDefs and
385 similar directives, it is possible to reduce the client-specific part of
386 a configuration to a minimum. The client-specific directives can be
387 prepared according to a standard template and dropped into a known
388 directory. However it is still necessary to add a line to the "master"
389 (bacula-dir.conf) referencing each new file. This exposes the master to
390 unnecessary risk of accidental mistakes and makes automation of adding
391 new client-confs, more difficult (it is easier to automate dropping a
392 file into a dir, than rewriting an existing file). Ken has previously
393 made a convincing argument for NOT including Bacula's core configuration
394 in an RDBMS, but I believe that the present request is a reasonable
395 extension to the current "flat-file-based" configuration philosophy.
397 Notes: There is NO need for any special syntax to these files. They should
398 contain standard directives which are simply "inlined" to the parent
399 file as already happens when you explicitly reference an external file.
401 Notes: (kes) this can already be done with scripting
402 From: John Jorgensen <jorgnsn@lcd.uregina.ca>
403 The bacula-dir.conf at our site contains these lines:
406 # Include subfiles associated with configuration of clients.
407 # They define the bulk of the Clients, Jobs, and FileSets.
409 @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'"
411 and when we get a new client, we just put its configuration into
412 a new file called something like:
414 /etc/bacula/clientdefs/clientname.conf
417 Item 17: Multiple threads in file daemon for the same job
418 Date: 27 November 2005
419 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
422 What: I want the file daemon to start multiple threads for a backup
423 job so the fastest possible backup can be made.
425 The file daemon could parse the FileSet information and start
426 one thread for each File entry located on a separate
429 A confiuration option in the job section should be used to
430 enable or disable this feature. The confgutration option could
431 specify the maximum number of threads in the file daemon.
433 If the theads could spool the data to separate spool files
434 the restore process will not be much slower.
436 Why: Multiple concurrent backups of a large fileserver with many
437 disks and controllers will be much faster.
440 Item 18: Possibilty to schedule Jobs on last Friday of the month
441 Origin: Carsten Menke <bootsy52 at gmx dot net>
445 What: Currently if you want to run your monthly Backups on the last
446 Friday of each month this is only possible with workarounds (e.g
447 scripting) (As some months got 4 Fridays and some got 5 Fridays)
448 The same is true if you plan to run your yearly Backups on the
449 last Friday of the year. It would be nice to have the ability to
450 use the builtin scheduler for this.
452 Why: In many companies the last working day of the week is Friday (or
453 Saturday), so to get the most data of the month onto the monthly
454 tape, the employees are advised to insert the tape for the
455 monthly backups on the last friday of the month.
457 Notes: To give this a complete functionality it would be nice if the
458 "first" and "last" Keywords could be implemented in the
459 scheduler, so it is also possible to run monthy backups at the
460 first friday of the month and many things more. So if the syntax
461 would expand to this {first|last} {Month|Week|Day|Mo-Fri} of the
462 {Year|Month|Week} you would be able to run really flexible jobs.
464 To got a certain Job run on the last Friday of the Month for example
467 Run = pool=Monthly last Fri of the Month at 23:50
471 Run = pool=Yearly last Fri of the Year at 23:50
473 ## Certain Jobs the last Week of a Month
475 Run = pool=LastWeek last Week of the Month at 23:50
477 ## Monthly Backup on the last day of the month
479 Run = pool=Monthly last Day of the Month at 23:50
482 Item 19: Include timestamp of job launch in "stat clients" output
483 Origin: Mark Bergman <mark.bergman@uphs.upenn.edu>
484 Date: Tue Aug 22 17:13:39 EDT 2006
487 What: The "stat clients" command doesn't include any detail on when
488 the active backup jobs were launched.
490 Why: Including the timestamp would make it much easier to decide whether
491 a job is running properly.
493 Notes: It may be helpful to have the output from "stat clients" formatted
494 more like that from "stat dir" (and other commands), in a column
495 format. The per-client information that's currently shown (level,
496 client name, JobId, Volume, pool, device, Files, etc.) is good, but
497 somewhat hard to parse (both programmatically and visually),
498 particularly when there are many active clients.
501 Item 21: Message mailing based on backup types
502 Origin: Evan Kaufman <evan.kaufman@gmail.com>
503 Date: January 6, 2006
506 What: In the "Messages" resource definitions, allowing messages
507 to be mailed based on the type (backup, restore, etc.) and level
508 (full, differential, etc) of job that created the originating
511 Why: It would, for example, allow someone's boss to be emailed
512 automatically only when a Full Backup job runs, so he can
513 retrieve the tapes for offsite storage, even if the IT dept.
514 doesn't (or can't) explicitly notify him. At the same time, his
515 mailbox wouldnt be filled by notifications of Verifies, Restores,
516 or Incremental/Differential Backups (which would likely be kept
519 Notes: One way this could be done is through additional message types, for
523 # email the boss only on full system backups
524 Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
526 # email us only when something breaks
527 MailOnError = itdept@mycompany.com = all
530 Notes: Kern: This should be rather trivial to implement.
533 Item 22: Ability to import/export Bacula database entities
538 What: Create a Bacula ASCII SQL database independent format that permits
539 importing and exporting database catalog Job entities.
541 Why: For achival, database clustering, tranfer to other databases
544 Notes: Job selection should be by Job, time, Volume, Client, Pool and possibly
548 Item 24: Implementation of running Job speed limit.
549 Origin: Alex F, alexxzell at yahoo dot com
550 Date: 29 January 2009
552 What: I noticed the need for an integrated bandwidth limiter for
553 running jobs. It would be very useful just to specify another
554 field in bacula-dir.conf, like speed = how much speed you wish
555 for that specific job to run at
557 Why: Because of a couple of reasons. First, it's very hard to implement a
558 traffic shaping utility and also make it reliable. Second, it is very
559 uncomfortable to have to implement these apps to, let's say 50 clients
560 (including desktops, servers). This would also be unreliable because you
561 have to make sure that the apps are properly working when needed; users
562 could also disable them (accidentally or not). It would be very useful
563 to provide Bacula this ability. All information would be centralized,
564 you would not have to go to 50 different clients in 10 different
565 locations for configuration; eliminating 3rd party additions help in
566 establishing efficiency. Would also avoid bandwidth congestion,
567 especially where there is little available.
570 Item 25: Add an override in Schedule for Pools based on backup types
572 Origin: Chad Slater <chad.slater@clickfox.com>
575 What: Adding a FullStorage=BigTapeLibrary in the Schedule resource
576 would help those of us who use different storage devices for different
577 backup levels cope with the "auto-upgrade" of a backup.
579 Why: Assume I add several new devices to be backed up, i.e. several
580 hosts with 1TB RAID. To avoid tape switching hassles, incrementals are
581 stored in a disk set on a 2TB RAID. If you add these devices in the
582 middle of the month, the incrementals are upgraded to "full" backups,
583 but they try to use the same storage device as requested in the
584 incremental job, filling up the RAID holding the differentials. If we
585 could override the Storage parameter for full and/or differential
586 backups, then the Full job would use the proper Storage device, which
587 has more capacity (i.e. a 8TB tape library.
590 Item 26: Automatic promotion of backup levels based on backup size
591 Date: 19 January 2006
592 Origin: Adam Thornton <athornton@sinenomine.net>
595 What: Other backup programs have a feature whereby it estimates the space
596 that a differential, incremental, and full backup would take. If
597 the difference in space required between the scheduled level and the
598 next level up is beneath some user-defined critical threshold, the
599 backup level is bumped to the next type. Doing this minimizes the
600 number of volumes necessary during a restore, with a fairly minimal
601 cost in backup media space.
603 Why: I know at least one (quite sophisticated and smart) user for whom the
604 absence of this feature is a deal-breaker in terms of using Bacula;
605 if we had it it would eliminate the one cool thing other backup
606 programs can do and we can't (at least, the one cool thing I know
610 Item 27: Allow inclusion/exclusion of files in a fileset by creation/mod times
611 Origin: Evan Kaufman <evan.kaufman@gmail.com>
612 Date: January 11, 2006
615 What: In the vein of the Wild and Regex directives in a Fileset's
616 Options, it would be helpful to allow a user to include or exclude
617 files and directories by creation or modification times.
619 You could factor the Exclude=yes|no option in much the same way it
620 affects the Wild and Regex directives. For example, you could exclude
621 all files modified before a certain date:
625 Modified Before = ####
628 Or you could exclude all files created/modified since a certain date:
632 Created Modified Since = ####
635 The format of the time/date could be done several ways, say the number
636 of seconds since the epoch:
637 1137008553 = Jan 11 2006, 1:42:33PM # result of `date +%s`
639 Or a human readable date in a cryptic form:
640 20060111134233 = Jan 11 2006, 1:42:33PM # YYYYMMDDhhmmss
642 Why: I imagine a feature like this could have many uses. It would
643 allow a user to do a full backup while excluding the base operating
644 system files, so if I installed a Linux snapshot from a CD yesterday,
645 I'll *exclude* all files modified *before* today. If I need to
646 recover the system, I use the CD I already have, plus the tape backup.
647 Or if, say, a Windows client is hit by a particularly corrosive
648 virus, and I need to *exclude* any files created/modified *since* the
651 Notes: Of course, this feature would work in concert with other
652 in/exclude rules, and wouldnt override them (or each other).
654 Notes: The directives I'd imagine would be along the lines of
655 "[Created] [Modified] [Before|Since] = <date>".
656 So one could compare against 'ctime' and/or 'mtime', but ONLY 'before'
660 Item 28: Archival (removal) of User Files to Tape
662 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
665 What: The ability to archive data to storage based on certain parameters
666 such as age, size, or location. Once the data has been written to
667 storage and logged it is then pruned from the originating
668 filesystem. Note! We are talking about user's files and not
671 Why: This would allow fully automatic storage management which becomes
672 useful for large datastores. It would also allow for auto-staging
673 from one media type to another.
675 Example 1) Medical imaging needs to store large amounts of data.
676 They decide to keep data on their servers for 6 months and then put
677 it away for long term storage. The server then finds all files
678 older than 6 months writes them to tape. The files are then removed
681 Example 2) All data that hasn't been accessed in 2 months could be
682 moved from high-cost, fibre-channel disk storage to a low-cost
683 large-capacity SATA disk storage pool which doesn't have as quick of
684 access time. Then after another 6 months (or possibly as one
685 storage pool gets full) data is migrated to Tape.
688 Item 29: An option to operate on all pools with update vol parameters
689 Origin: Dmitriy Pinchukov <absh@bossdev.kiev.ua>
691 Status: Patch made by Nigel Stepp
693 What: When I do update -> Volume parameters -> All Volumes
694 from Pool, then I have to select pools one by one. I'd like
695 console to have an option like "0: All Pools" in the list of
698 Why: I have many pools and therefore unhappy with manually
699 updating each of them using update -> Volume parameters -> All
700 Volumes from Pool -> pool #.
703 Item 30: Automatic disabling of devices
705 Origin: Peter Eriksson <peter at ifm.liu dot se>
708 What: After a configurable amount of fatal errors with a tape drive
709 Bacula should automatically disable further use of a certain
710 tape drive. There should also be "disable"/"enable" commands in
713 Why: On a multi-drive jukebox there is a possibility of tape drives
714 going bad during large backups (needing a cleaning tape run,
715 tapes getting stuck). It would be advantageous if Bacula would
716 automatically disable further use of a problematic tape drive
717 after a configurable amount of errors has occurred.
719 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
720 where tapes occasionally get stuck inside the drive. Bacula will
721 notice that the "mtx-changer" command will fail and then fail
722 any backup jobs trying to use that drive. However, it will still
723 keep on trying to run new jobs using that drive and fail -
724 forever, and thus failing lots and lots of jobs... Since we have
725 many drives Bacula could have just automatically disabled
726 further use of that drive and used one of the other ones
730 Item 32: Ability to defer Batch Insert to a later time
735 What: Instead of doing a Job Batch Insert at the end of the Job
736 which might create resource contention with lots of Job,
737 defer the insert to a later time.
739 Why: Permits to focus on getting the data on the Volume and
740 putting the metadata into the Catalog outside the backup
743 Notes: Will use the proposed Bacula ASCII database import/export
744 format (i.e. dependent on the import/export entities project).
747 Item 33: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
748 Origin: Bastian Friedrich <bastian.friedrich@collax.com>
752 What: SD has a "Maximum Volume Size" statement, which is deprecated and
753 superseded by the Pool resource statement "Maximum Volume Bytes".
754 It would be good if either statement could be used in Storage
757 Why: Pools do not have to be restricted to a single storage type/device;
758 thus, it may be impossible to define Maximum Volume Bytes in the
759 Pool resource. The old MaxVolSize statement is deprecated, as it
760 is SD side only. I am using the same pool for different devices.
762 Notes: State of idea currently unknown. Storage resources in the dir
763 config currently translate to very slim catalog entries; these
764 entries would require extensions to implement what is described
765 here. Quite possibly, numerous other statements that are currently
766 available in Pool resources could be used in Storage resources too
770 Item 34: Enable persistent naming/number of SQL queries
776 Change the parsing of the query.sql file and the query command so that
777 queries are named/numbered by a fixed value, not their order in the
782 One of the real strengths of bacula is the ability to query the
783 database, and the fact that complex queries can be saved and
784 referenced from a file is very powerful. However, the choice
785 of query (both for interactive use, and by scripting input
786 to the bconsole command) is completely dependent on the order
787 within the query.sql file. The descriptve labels are helpful for
788 interactive use, but users become used to calling a particular
789 query "by number", or may use scripts to execute queries. This
790 presents a problem if the number or order of queries in the file
793 If the query.sql file used the numeric tags as a real value (rather
794 than a comment), then users could have a higher confidence that they
795 are executing the intended query, that their local changes wouldn't
796 conflict with future bacula upgrades.
798 For scripting, it's very important that the intended query is
799 what's actually executed. The current method of parsing the
800 query.sql file discourages scripting because the addition or
801 deletion of queries within the file will require corresponding
802 changes to scripts. It may not be obvious to users that deleting
803 query "17" in the query.sql file will require changing all
804 references to higher numbered queries. Similarly, when new
805 bacula distributions change the number of "official" queries,
806 user-developed queries cannot simply be appended to the file
807 without also changing any references to those queries in scripts
808 or procedural documentation, etc.
810 In addition, using fixed numbers for queries would encourage more
811 user-initiated development of queries, by supporting conventions
814 queries numbered 1-50 are supported/developed/distributed by
815 with official bacula releases
817 queries numbered 100-200 are community contributed, and are
818 related to media management
820 queries numbered 201-300 are community contributed, and are
821 related to checksums, finding duplicated files across
822 different backups, etc.
824 queries numbered 301-400 are community contributed, and are
825 related to backup statistics (average file size, size per
826 client per backup level, time for all clients by backup level,
827 storage capacity by media type, etc.)
829 queries numbered 500-999 are locally created
832 Alternatively, queries could be called by keyword (tag), rather
836 Item 36: Bacula Dir, FD and SD to support proxies
837 Origin: Karl Grindley @ MIT Lincoln Laboratory <kgrindley at ll dot mit dot edu>
841 What: Support alternate methods for nailing up a TCP session such
842 as SOCKS5, SOCKS4 and HTTP (CONNECT) proxies. Such a feature
843 would allow tunneling of bacula traffic in and out of proxied
846 Why: Currently, bacula is architected to only function on a flat network, with
847 no barriers or limitations. Due to the large configuration states of
848 any network and the infinite configuration where file daemons and
849 storage daemons may sit in relation to one another, bacula often is
850 not usable on a network where filtered or air-gaped networks exist.
851 While often solutions such as ACL modifications to firewalls or port
852 redirection via SNAT or DNAT will solve the issue, often however,
853 these solutions are not adequate or not allowed by hard policy.
855 In an air-gapped network with only a highly locked down proxy services
856 are provided (SOCKS4/5 and/or HTTP and/or SSH outbound) ACLs or
857 iptable rules will not work.
859 Notes: Director resource tunneling: This configuration option to utilize a
860 proxy to connect to a client should be specified in the client
861 resource Client resource tunneling: should be configured in the client
862 resource in the director config file? Or configured on the bacula-fd
863 configuration file on the fd host itself? If the ladder, this would
864 allow only certain clients to use a proxy, where others do not when
865 establishing the TCP connection to the storage server.
867 Also worth noting, there are other 3rd party, light weight apps that
868 could be utilized to bootstrap this. Instead of sockifing bacula
869 itself, use an external program to broker proxy authentication, and
870 connection to the remote host. OpenSSH does this by using the
871 "ProxyCommand" syntax in the client configuration and uses stdin and
872 stdout to the command. Connect.c is a very popular one.
873 (http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html).
874 One could also possibly use stunnel, netcat, etc.
877 Item 37: Add Minumum Spool Size directive
879 Origin: Frank Sweetser <fs@wpi.edu>
881 What: Add a new SD directive, "minimum spool size" (or similar). This
882 directive would specify a minimum level of free space available for
883 spooling. If the unused spool space is less than this level, any
884 new spooling requests would be blocked as if the "maximum spool
885 size" threshold had bee reached. Already spooling jobs would be
886 unaffected by this directive.
888 Why: I've been bitten by this scenario a couple of times:
890 Assume a maximum spool size of 100M. Two concurrent jobs, A and B,
891 are both running. Due to timing quirks and previously running jobs,
892 job A has used 99.9M of space in the spool directory. While A is
893 busy despooling to disk, B is happily using the remaining 0.1M of
894 spool space. This ends up in a spool/despool sequence every 0.1M of
895 data. In addition to fragmenting the data on the volume far more
896 than was necessary, in larger data sets (ie, tens or hundreds of
897 gigabytes) it can easily produce multi-megabyte report emails!
900 Item 38: Backup and Restore of Windows Encrypted Files using Win raw encryption
901 Origin: Michael Mohr, SAG Mohr.External@infineon.com
902 Date: 22 February 2008
903 Origin: Alex Ehrlich (Alex.Ehrlich-at-mail.ee)
907 What: Make it possible to backup and restore Encypted Files from and to
908 Windows systems without the need to decrypt it by using the raw
909 encryption functions API (see:
910 http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
911 that is provided for that reason by Microsoft.
912 If a file ist encrypted could be examined by evaluating the
913 FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes
915 For each file backed up or restored by FD on Windows, check if
916 the file is encrypted; if so then use OpenEncryptedFileRaw,
917 ReadEncryptedFileRaw, WriteEncryptedFileRaw,
918 CloseEncryptedFileRaw instead of BackupRead and BackupWrite
921 Why: Without the usage of this interface the fd-daemon running
922 under the system account can't read encypted Files because
923 the key needed for the decrytion is missed by them. As a result
924 actually encrypted files are not backed up
925 by bacula and also no error is shown while missing these files.
927 Notes: Using xxxEncryptedFileRaw API would allow to backup and
928 restore EFS-encrypted files without decrypting their data.
929 Note that such files cannot be restored "portably" (at least,
930 easily) but they would be restoreable to a different (or
931 reinstalled) Win32 machine; the restore would require setup
932 of a EFS recovery agent in advance, of course, and this shall
933 be clearly reflected in the documentation, but this is the
934 normal Windows SysAdmin's business.
935 When "portable" backup is requested the EFS-encrypted files
936 shall be clearly reported as errors.
937 See MSDN on the "Backup and Restore of Encrypted Files" topic:
938 http://msdn.microsoft.com/en-us/library/aa363783.aspx
939 Maybe the EFS support requires a new flag in the database for
941 Unfortunately, the implementation is not as straightforward as
942 1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
943 requiring some FD code rewrite to work with
944 encrypted-file-related callback functions.
947 Item 39: Implement an interface between Bacula and Storage clould like Amazon's S3.
949 Origin: Soren Hansen <soren@ubuntu.com>
951 What: Enable the storage daemon to store backup data on Amazon's
954 Why: Amazon's S3 is a cheap way to store data off-site.
956 Notes: If we configure the Pool to put only one job per volume (they don't
957 support append operation), and the volume size isn't to big (100MB?),
958 it should be easy to adapt the disk-changer script to add get/put
959 procedure with curl. So, the data would be safetly copied during the
962 Cloud should be only used with Copy jobs, users should always have
963 a copy of their data on their site.
965 We should also think to have our own cache, trying always to have
966 cloud volume on the local disk. (I don't know if users want to store
967 100GB on cloud, so it shouldn't be a disk size problem). For example,
968 if bacula want to recycle a volume, it will start by downloading the
969 file to truncate it few seconds later, if we can avoid that...
971 Item 40: Convert Bacula existing tray monitor on Windows to a stand alone program
976 What: Separate Win32 tray monitor to be a separate program.
978 Why: Vista does not allow SYSTEM services to interact with the
979 desktop, so the current tray monitor does not work on Vista
982 Notes: Requires communicating with the FD via the network (simulate
983 a console connection).
987 ========= End items voted on May 2009 ==================
989 ========= New items after last vote ====================
991 Item 1: Relabel disk volume after recycling
992 Origin: Pasi Kärkkäinen <pasik@iki.fi>
994 Status: Not implemented yet, no code written.
996 What: The ability to relabel the disk volume (and thus rename the file on the
997 disk) after it has been recycled. Useful when you have a single job
998 per disk volume, and you use a custom Label format, for example:
1000 "${Client}-${Level}-${NumVols:p/4/0/r}-${Year}_${Month}_${Day}-${Hour}_${Minute}"
1002 Why: Disk volumes in Bacula get the label/filename when they are used for the
1003 first time. If you use recycling and custom label format like above,
1004 the disk volume name doesn't match the contents after it has been
1005 recycled. This feature makes it possible to keep the label/filename
1006 in sync with the content and thus makes it easy to check/monitor the
1007 backups from the shell and/or normal file management tools, because
1008 the filenames of the disk volumes match the content.
1010 Notes: The configuration option could be "Relabel after Recycling = Yes".
1012 Item n: Command that releases all drives in an autochanger
1013 Origin: Blake Dunlap (blake@nxs.net)
1017 What: It would be nice if there was a release command that
1018 would release all drives in an autochanger instead of having to
1019 do each one in turn.
1021 Why: It can take some time for a release to occur, and the
1022 commands must be given for each drive in turn, which can quicky
1023 scale if there are several drives in the library. (Having to
1024 watch the console, to give each command can waste a good bit of
1025 time when you start getting into the 16 drive range when the
1026 tapes can take up to 3 minutes to eject each)
1028 Notes: Due to the way some autochangers/libraries work, you
1029 cannot assume that new tapes inserted will go into slots that are
1030 not currently believed to be in use by bacula (the tape from that
1031 slot is in a drive). This would make any changes in
1032 configuration quicker/easier, as all drives need to be released
1033 before any modifications to slots.
1035 Item n: Run bscan on a remote storage daemon from within bconsole.
1036 Date: 07 October 2009
1037 Origin: Graham Keeling <graham@equiinet.com>
1040 What: The ability to be able to run bscan on a remote storage daemon from
1041 within bconsole in order to populate your catalog.
1043 Why: Currently, it seems you have to:
1044 a) log in to a console on the remote machine
1045 b) figure out where the storage daemon config file is
1046 c) figure out the storage device from the config file
1047 d) figure out the catalog IP address
1048 e) figure out the catalog port
1049 f) open the port on the catalog firewall
1050 g) configure the catalog database to accept connections from the
1052 h) build a 'bscan' command from (b)-(e) above and run it
1053 It would be much nicer to be able to type something like this into
1055 *bscan storage=<storage> device=<device> volume=<volume>
1057 *bscan storage=<storage> all
1058 It seems to me that the scan could also do a better job than the
1059 external bscan program currently does. It would possibly be able to
1060 deduce some extra details, such as the catalog StorageId for the
1063 Notes: (Kern). If you need to do a bscan, you have done something wrong,
1064 so this functionality should not need to be integrated into the
1065 the Storage daemon. However, I am not opposed to someone implementing
1066 this feature providing that all the code is in a shared object (or dll)
1067 and does not add significantly to the size of the Storage daemon. In
1068 addition, the code should be written in a way such that the same source
1069 code is used in both the bscan program and the Storage daemon to avoid
1070 adding a lot of new code that must be maintained by the project.
1072 Item n: Implement a Migration job type that will create a reverse
1073 incremental (or decremental) backup from two existing full backups.
1074 Date: 05 October 2009
1075 Origin: Griffith College Dublin. Some sponsorship available.
1076 Contact: Gavin McCullagh <gavin.mccullagh@gcd.ie>
1079 What: The ability to take two full backup jobs and derive a reverse
1080 incremental backup from them. The older full backup data may then
1083 Why: Long-term backups based on keeping full backups can be expensive in
1084 media. In many cases (eg a NAS), as the client accumulates files
1085 over months and years, the same file will be duplicated unchanged,
1086 across many media and datasets. Eg, Less than 10% (and
1087 shrinking) of our monthly full mail server backup is new files,
1088 the other 90% is also in the previous full backup.
1089 Regularly converting the oldest full backup into a reverse
1090 incremental backup allows the admin to keep access to old backup
1091 jobs, but remove all of the duplicated files, freeing up media.
1093 Notes: This feature was previously discussed on the bacula-devel list
1094 here: http://www.mail-archive.com/bacula-devel@lists.sourceforge.net/msg04962.html
1096 Item n: Job migration between different SDs
1097 Origin: Mariusz Czulada <manieq AT wp DOT eu>
1101 What: Allow to specify in migration job devices on Storage Daemon other then
1102 the one used for migrated jobs (possibly on different/distant host)
1104 Why: Sometimes we have more then one system which requires backup
1105 implementation. Often, these systems are functionally unrelated and
1106 placed in different locations. Having a big backup device (a tape
1107 library) in each location is not cost-effective. It would be much
1108 better to have one powerful enough tape library which could handle
1109 backups from all systems, assuming relatively fast and reliable WAN
1110 connections. In such architecture backups are done in service windows
1111 on local bacula servers, then migrated to central storage off the peak
1114 Notes: If migration to different SD is working, migration to the same SD, as
1115 now, could be done the same way (i mean 'localhost') to unify the
1118 Item n: Concurrent spooling and despooling withini a single job.
1120 Origin: Jesper Krogh <jesper@krogh.cc>
1122 What: When a job has spooling enabled and the spool area size is
1123 less than the total volumes size the storage daemon will:
1124 1) Spool to spool-area
1126 3) Go to 1 if more data to be backed up.
1128 Typical disks will serve data with a speed of 100MB/s when
1129 dealing with large files, network it typical capable of doing 115MB/s
1130 (GbitE). Tape drives will despool with 50-90MB/s (LTO3) 70-120MB/s
1131 (LTO4) depending on compression and data.
1133 As bacula currently works it'll hold back data from the client until
1134 de-spooling is done, now matter if the spool area can handle another
1135 block of data. Say given a FileSet of 4TB and a spool-area of 100GB and
1136 a Maximum Job Spool Size set to 50GB then above sequence could be
1137 changed to allow to spool to the other 50GB while despooling the first
1138 50GB and not holding back the client while doing it. As above numbers
1139 show, depending on tape-drive and disk-arrays this potentially leads to
1140 a cut of the backup-time of 50% for the individual jobs.
1142 Real-world example, backing up 112.6GB (large files) to LTO4 tapes
1143 (despools with ~75MB/s, data is gzipped on the remote filesystem.
1144 Maximum Job Spool Size = 8GB
1148 Elapsed time (total time): 46m 15s => 2775s
1149 Despooling time: 25m 41s => 1541s (55%)
1150 Spooling time: 20m 34s => 1234s (45%)
1151 Reported speed: 40.58MB/s
1152 Spooling speed: 112.6GB/1234s => 91.25MB/s
1153 Despooling speed: 112.6GB/1541s => 73.07MB/s
1155 So disk + net can "keep up" with the LTO4 drive (in this test)
1157 Prosed change would effectively make the backup run in the "despooling
1158 time" 1541s giving a reduction to 55% of the total run time.
1160 In the situation where the individual job cannot keep up with LTO-drive
1161 spooling enables efficient multiplexing of multiple concurrent jobs onto
1164 Why: When dealing with larger volumes the general utillization of the
1165 network/disk is important to maximize in order to be able to run a full
1166 backup over a weekend. Current work-around is to split the FileSet in
1167 smaller FileSet and Jobs but that leads to more configuration mangement
1168 and is harder to review for completeness. Subsequently it makes restores
1171 Item 1: Extend the verify code to make it possible to verify
1172 older jobs, not only the last one that has finished
1174 Origin: Ralf Gross (Ralf-Lists <at> ralfgross.de)
1175 Status: not implemented or documented
1177 What: At the moment a VolumeToCatalog job compares only the
1178 last job with the data in the catalog. It's not possible
1179 to compare the data (md5sums) of an older volume with the
1180 data in the catalog.
1182 Why: If a verify job fails, one has to immediately check the
1183 source of the problem, fix it and rerun the verify job.
1184 This has to happen before the next backup of the
1185 verified backup job starts.
1186 More important: It's not possible to check jobs that are
1187 kept for a long time (archiv). If a jobid could be
1188 specified for a verify job, older backups/tapes could be
1189 checked on a regular base.
1191 Notes: verify documentation:
1192 VolumeToCatalog: This level causes Bacula to read the file
1193 attribute data written to the Volume from the last Job [...]
1195 Verify Job = <Job-Resource-Name> If you run a verify job
1196 without this directive, the last job run will be compared
1197 with the catalog, which means that you must immediately
1198 follow a backup by a verify command. If you specify a Verify
1199 Job Bacula will find the last job with that name that ran [...]
1201 example bconsole verify dialog:
1204 JobName: VerifyServerXXX
1205 Level: VolumeToCatalog
1206 Client: ServerXXX-fd
1207 FileSet: ServerXXX-Vol1
1208 Pool: Full (From Job resource)
1209 Storage: Neo4100 (From Pool resource)
1210 Verify Job: ServerXXX-Vol1
1212 When: 2009-04-20 09:03:04
1214 OK to run? (yes/mod/no): m
1215 Parameters to modify:
1228 Item n: Separate "Storage" and "Device" in the bacula-dir.conf
1230 Origin: "James Harper" <james.harper@bendigoit.com.au>
1231 Status: not implemented or documented
1233 What: Separate "Storage" and "Device" in the bacula-dir.conf
1234 The resulting config would looks something like:
1237 Name = name_of_server
1238 Address = hostname/IP address
1240 Password = shh_its_a_secret
1241 Maximum Concurrent Jobs = 7
1245 Name = name_of_device
1246 Storage = name_of_server
1247 Device = name_of_device_on_sd
1248 Media Type = media_type
1249 Maximum Concurrent Jobs = 1
1252 Maximum Concurrent Jobs would be specified with a server and a device
1253 maximum, which would both be honoured by the director. Almost everything
1254 that mentions a 'Storage' would need to be changed to 'Device', although
1255 perhaps a 'Storage' would just be a synonym for 'Device' for backwards
1258 Why: If you have multiple Storage definitions pointing to different
1259 Devices in the same Storage daemon, the "status storage" command
1260 prompts for each different device, but they all give the same
1267 ========= Add new items above this line =================
1270 ============= Empty Feature Request form ===========
1271 Item n: One line summary ...
1272 Date: Date submitted
1273 Origin: Name and email of originator.
1276 What: More detailed explanation ...
1278 Why: Why it is important ...
1280 Notes: Additional notes or features (omit if not used)
1281 ============== End Feature Request form ==============
1284 ========== Items put on hold by Kern ============================
1287 ========== Items completed in version 5.0.0 ====================
1288 *Item 2: 'restore' menu: enter a JobId, automatically select dependents
1289 *Item 5: Deletion of disk Volumes when pruned (partial -- truncate when pruned)
1290 *Item 6: Implement Base jobs
1291 *Item 10: Restore from volumes on multiple storage daemons
1292 *Item 15: Enable/disable compression depending on storage device (disk/tape)
1293 *Item 20: Cause daemons to use a specific IP address to source communications
1294 *Item 23: "Maximum Concurrent Jobs" for drives when used with changer device
1295 *Item 31: List InChanger flag when doing restore.
1296 *Item 35: Port bat to Win32