3 Bacula Projects Roadmap
4 Status updated 8 August 2010
9 Item 1: Ability to restart failed jobs
11 Item* 3: NDMP backup/restore
12 Item 4: SAP backup/restore
13 Item 5: Oracle backup/restore
14 Item 6: Zimbra and Zarafa backup/restore
15 Item* 7: Include timestamp of job launch in "stat clients" output
16 Item 8: Include all conf files in specified directory
17 Item 9: Reduction of communications bandwidth for a backup
18 Item 10: Concurrent spooling and despooling within a single job.
19 Item 11: Start spooling even when waiting on tape
20 Item*12: Add ability to Verify any specified Job.
21 Item 13: Data encryption on storage daemon
22 Item 14: Possibilty to schedule Jobs on last Friday of the month
23 Item 15: Scheduling syntax that permits more flexibility and options
24 Item 16: Ability to defer Batch Insert to a later time
25 Item 17: Add MaxVolumeSize/MaxVolumeBytes to Storage resource
26 Item 18: Message mailing based on backup types
27 Item 19: Handle Windows Encrypted Files using Win raw encryption
28 Item 20: Job migration between different SDs
29 Item 19. Allow FD to initiate a backup
30 Item 21: Implement Storage daemon compression
31 Item 22: Ability to import/export Bacula database entities
32 Item*23: Implementation of running Job speed limit.
33 Item 24: Add an override in Schedule for Pools based on backup types
34 Item 25: Automatic promotion of backup levels based on backup size
35 Item 26: Allow FileSet inclusion/exclusion by creation/mod times
36 Item 27: Archival (removal) of User Files to Tape
37 Item 28: Ability to reconnect a disconnected comm line
38 Item 29: Multiple threads in file daemon for the same job
39 Item 30: Automatic disabling of devices
40 Item 31: Enable persistent naming/number of SQL queries
41 Item 32: Bacula Dir, FD and SD to support proxies
42 Item 33: Add Minumum Spool Size directive
43 Item 34: Command that releases all drives in an autochanger
44 Item 35: Run bscan on a remote storage daemon from within bconsole.
45 Item 36: Implement a Migration job type that will create a reverse
46 Item 37: Separate "Storage" and "Device" in the bacula-dir.conf
47 Item 38: Least recently used device selection for tape drives in autochanger.
48 Item 39: Implement a Storage device like Amazon's S3.
49 Item*40: Convert tray monitor on Windows to a stand alone program
50 Item 41: Improve Bacula's tape and drive usage and cleaning management
51 Item 42: Relabel disk volume after recycling
53 Item 1: Ability to restart failed jobs
58 What: Often jobs fail because of a communications line drop or max run time,
59 cancel, or some other non-critical problem. Currrently any data
60 saved is lost. This implementation should modify the Storage daemon
61 so that it saves all the files that it knows are completely backed
64 The jobs should then be marked as incomplete and a subsequent
65 Incremental Accurate backup will then take into account all the
68 Why: Avoids backuping data already saved.
70 Notes: Requires Accurate to restart correctly. Must completed have a minimum
71 volume of data or files stored on Volume before enabling.
78 What: Various ideas for redesigns planned for the SD:
79 1. One thread per drive
80 2. Design a class structure for all objects in the SD.
81 3. Make Device into C++ classes for each device type
82 4. Make Device have a proxy (front end intercept class) that will permit control over locking and changing the real device pointer. It can also permit delaying opening, so that we can adapt to having another program that tells us the Archive device name.
83 5. Allow plugins to create new on the fly devices
84 6. Separate SD volume manager
85 7. Volume manager tells Bacula what drive or device to use for a given volume
87 Why: It will simplify the SD, make it more modular, reduce locking
88 conflicts, and allow multiple buffer backups.
91 Item 3: NDMP backup/restore
93 Origin: Bacula Systems
94 Status: Enterprise only if implemented by Bacula Systems
96 What: Backup/restore via NDMP -- most important NetApp compatibility
100 Item 4: SAP backup/restore
102 Origin: Bacula Systems
103 Status: Enterprise only if implemented by Bacula Systems
105 What: Backup/restore SAP databases (MaxDB, Oracle, possibly DB2)
109 Item 5: Oracle backup/restore
111 Origin: Bacula Systems
112 Status: Enterprise only if implemented by Bacula Systems
114 What: Backup/restore Oracle databases
117 Item 6: Zimbra and Zarafa backup/restore
119 Origin: Bacula Systems
120 Status: Enterprise only if implemented by Bacula Systems
122 What: Backup/restore for Zimbra and Zarafa
126 Item 7: Include timestamp of job launch in "stat clients" output
127 Origin: Mark Bergman <mark.bergman@uphs.upenn.edu>
128 Date: Tue Aug 22 17:13:39 EDT 2006
131 What: The "stat clients" command doesn't include any detail on when
132 the active backup jobs were launched.
134 Why: Including the timestamp would make it much easier to decide whether
135 a job is running properly.
137 Notes: It may be helpful to have the output from "stat clients" formatted
138 more like that from "stat dir" (and other commands), in a column
139 format. The per-client information that's currently shown (level,
140 client name, JobId, Volume, pool, device, Files, etc.) is good, but
141 somewhat hard to parse (both programmatically and visually),
142 particularly when there are many active clients.
145 Item 8: Include all conf files in specified directory
146 Date: 18 October 2008
147 Origin: Database, Lda. Maputo, Mozambique
148 Contact:Cameron Smith / cameron.ord@database.co.mz
151 What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every
152 time Bacula Director restarts or reloads, it will walk the given
153 directory (non-recursively) and include the contents of any files
154 therein, as though they were appended to bacula-dir.conf
156 Why: Permits simplified and safer configuration for larger installations with
157 many client PCs. Currently, through judicious use of JobDefs and
158 similar directives, it is possible to reduce the client-specific part of
159 a configuration to a minimum. The client-specific directives can be
160 prepared according to a standard template and dropped into a known
161 directory. However it is still necessary to add a line to the "master"
162 (bacula-dir.conf) referencing each new file. This exposes the master to
163 unnecessary risk of accidental mistakes and makes automation of adding
164 new client-confs, more difficult (it is easier to automate dropping a
165 file into a dir, than rewriting an existing file). Ken has previously
166 made a convincing argument for NOT including Bacula's core configuration
167 in an RDBMS, but I believe that the present request is a reasonable
168 extension to the current "flat-file-based" configuration philosophy.
170 Notes: There is NO need for any special syntax to these files. They should
171 contain standard directives which are simply "inlined" to the parent
172 file as already happens when you explicitly reference an external file.
174 Notes: (kes) this can already be done with scripting
175 From: John Jorgensen <jorgnsn@lcd.uregina.ca>
176 The bacula-dir.conf at our site contains these lines:
179 # Include subfiles associated with configuration of clients.
180 # They define the bulk of the Clients, Jobs, and FileSets.
182 @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'"
184 and when we get a new client, we just put its configuration into
185 a new file called something like:
187 /etc/bacula/clientdefs/clientname.conf
192 Item 9: Reduction of communications bandwidth for a backup
193 Date: 14 October 2008
194 Origin: Robin O'Leary (Equiinet)
197 What: Using rdiff techniques, Bacula could significantly reduce
198 the network data transfer volume to do a backup.
200 Why: Faster backup across the Internet
202 Notes: This requires retaining certain data on the client during a Full
203 backup that will speed up subsequent backups.
206 Item 10: Concurrent spooling and despooling within a single job.
208 Origin: Jesper Krogh <jesper@krogh.cc>
210 What: When a job has spooling enabled and the spool area size is
211 less than the total volumes size the storage daemon will:
212 1) Spool to spool-area
214 3) Go to 1 if more data to be backed up.
216 Typical disks will serve data with a speed of 100MB/s when
217 dealing with large files, network it typical capable of doing 115MB/s
218 (GbitE). Tape drives will despool with 50-90MB/s (LTO3) 70-120MB/s
219 (LTO4) depending on compression and data.
221 As bacula currently works it'll hold back data from the client until
222 de-spooling is done, now matter if the spool area can handle another
223 block of data. Say given a FileSet of 4TB and a spool-area of 100GB and
224 a Maximum Job Spool Size set to 50GB then above sequence could be
225 changed to allow to spool to the other 50GB while despooling the first
226 50GB and not holding back the client while doing it. As above numbers
227 show, depending on tape-drive and disk-arrays this potentially leads to
228 a cut of the backup-time of 50% for the individual jobs.
230 Real-world example, backing up 112.6GB (large files) to LTO4 tapes
231 (despools with ~75MB/s, data is gzipped on the remote filesystem.
232 Maximum Job Spool Size = 8GB
236 Elapsed time (total time): 46m 15s => 2775s
237 Despooling time: 25m 41s => 1541s (55%)
238 Spooling time: 20m 34s => 1234s (45%)
239 Reported speed: 40.58MB/s
240 Spooling speed: 112.6GB/1234s => 91.25MB/s
241 Despooling speed: 112.6GB/1541s => 73.07MB/s
243 So disk + net can "keep up" with the LTO4 drive (in this test)
245 Prosed change would effectively make the backup run in the "despooling
246 time" 1541s giving a reduction to 55% of the total run time.
248 In the situation where the individual job cannot keep up with LTO-drive
249 spooling enables efficient multiplexing of multiple concurrent jobs onto
252 Why: When dealing with larger volumes the general utillization of the
253 network/disk is important to maximize in order to be able to run a full
254 backup over a weekend. Current work-around is to split the FileSet in
255 smaller FileSet and Jobs but that leads to more configuration mangement
256 and is harder to review for completeness. Subsequently it makes restores
261 Item 11: Start spooling even when waiting on tape
262 Origin: Tobias Barth <tobias.barth@web-arts.com>
266 What: If a job can be spooled to disk before writing it to tape, it should
267 be spooled immediately. Currently, bacula waits until the correct
268 tape is inserted into the drive.
270 Why: It could save hours. When bacula waits on the operator who must insert
271 the correct tape (e.g. a new tape or a tape from another media
272 pool), bacula could already prepare the spooled data in the spooling
273 directory and immediately start despooling when the tape was
274 inserted by the operator.
276 2nd step: Use 2 or more spooling directories. When one directory is
277 currently despooling, the next (on different disk drives) could
278 already be spooling the next data.
280 Notes: I am using bacula 2.2.8, which has none of those features
284 Item 12: Add ability to Verify any specified Job.
285 Date: 17 January 2008
286 Origin: portrix.net Hamburg, Germany.
287 Contact: Christian Sabelmann
288 Status: Can use jobid= in run command to select an old job
291 The ability to tell Bacula which Job should verify instead of
292 automatically verify just the last one.
295 It is sad that such a powerfull feature like Verify Jobs
296 (VolumeToCatalog) is restricted to be used only with the last backup Job
297 of a client. Actual users who have to do daily Backups are forced to
298 also do daily Verify Jobs in order to take advantage of this useful
299 feature. This Daily Verify after Backup conduct is not always desired
300 and Verify Jobs have to be sometimes scheduled. (Not necessarily
301 scheduled in Bacula). With this feature Admins can verify Jobs once a
302 Week or less per month, selecting the Jobs they want to verify. This
303 feature is also not to difficult to implement taking in account older bug
304 reports about this feature and the selection of the Job to be verified.
306 Notes: For the verify Job, the user could select the Job to be verified
307 from a List of the latest Jobs of a client. It would also be possible to
308 verify a certain volume. All of these would naturaly apply only for
309 Jobs whose file information are still in the catalog.
312 Item 13: Data encryption on storage daemon
313 Origin: Tobias Barth <tobias.barth at web-arts.com>
314 Date: 04 February 2009
317 What: The storage demon should be able to do the data encryption that can
318 currently be done by the file daemon.
320 Why: This would have 2 advantages:
321 1) one could encrypt the data of unencrypted tapes by doing a
323 2) the storage daemon would be the only machine that would have
324 to keep the encryption keys.
327 As an addendum to the feature request, here are some crypto
328 implementation details I wrote up regarding SD-encryption back in Jan
330 http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html
334 Item 14: Possibilty to schedule Jobs on last Friday of the month
335 Origin: Carsten Menke <bootsy52 at gmx dot net>
339 What: Currently if you want to run your monthly Backups on the last
340 Friday of each month this is only possible with workarounds (e.g
341 scripting) (As some months got 4 Fridays and some got 5 Fridays)
342 The same is true if you plan to run your yearly Backups on the
343 last Friday of the year. It would be nice to have the ability to
344 use the builtin scheduler for this.
346 Why: In many companies the last working day of the week is Friday (or
347 Saturday), so to get the most data of the month onto the monthly
348 tape, the employees are advised to insert the tape for the
349 monthly backups on the last friday of the month.
351 Notes: To give this a complete functionality it would be nice if the
352 "first" and "last" Keywords could be implemented in the
353 scheduler, so it is also possible to run monthy backups at the
354 first friday of the month and many things more. So if the syntax
355 would expand to this {first|last} {Month|Week|Day|Mo-Fri} of the
356 {Year|Month|Week} you would be able to run really flexible jobs.
358 To got a certain Job run on the last Friday of the Month for example
361 Run = pool=Monthly last Fri of the Month at 23:50
365 Run = pool=Yearly last Fri of the Year at 23:50
367 ## Certain Jobs the last Week of a Month
369 Run = pool=LastWeek last Week of the Month at 23:50
371 ## Monthly Backup on the last day of the month
373 Run = pool=Monthly last Day of the Month at 23:50
375 Item 15: Scheduling syntax that permits more flexibility and options
376 Date: 15 December 2006
377 Origin: Gregory Brauer (greg at wildbrain dot com) and
378 Florian Schnabel <florian.schnabel at docufy dot de>
381 What: Currently, Bacula only understands how to deal with weeks of the
382 month or weeks of the year in schedules. This makes it impossible
383 to do a true weekly rotation of tapes. There will always be a
384 discontinuity that will require disruptive manual intervention at
385 least monthly or yearly because week boundaries never align with
386 month or year boundaries.
388 A solution would be to add a new syntax that defines (at least)
389 a start timestamp, and repetition period.
391 An easy option to skip a certain job on a certain date.
394 Why: Rotated backups done at weekly intervals are useful, and Bacula
395 cannot currently do them without extensive hacking.
397 You could then easily skip tape backups on holidays. Especially
398 if you got no autochanger and can only fit one backup on a tape
399 that would be really handy, other jobs could proceed normally
400 and you won't get errors that way.
403 Notes: Here is an example syntax showing a 3-week rotation where full
404 Backups would be performed every week on Saturday, and an
405 incremental would be performed every week on Tuesday. Each
406 set of tapes could be removed from the loader for the following
407 two cycles before coming back and being reused on the third
408 week. Since the execution times are determined by intervals
409 from a given point in time, there will never be any issues with
410 having to adjust to any sort of arbitrary time boundary. In
411 the example provided, I even define the starting schedule
412 as crossing both a year and a month boundary, but the run times
413 would be based on the "Repeat" value and would therefore happen
418 Name = "Week 1 Rotation"
419 #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
423 Start = 2006-12-30 01:00
427 #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
431 Start = 2007-01-02 01:00
438 Name = "Week 2 Rotation"
439 #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
443 Start = 2007-01-06 01:00
447 #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
451 Start = 2007-01-09 01:00
458 Name = "Week 3 Rotation"
459 #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
463 Start = 2007-01-13 01:00
467 #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
471 Start = 2007-01-16 01:00
477 Notes: Kern: I have merged the previously separate project of skipping
478 jobs (via Schedule syntax) into this.
481 Item 16: Ability to defer Batch Insert to a later time
486 What: Instead of doing a Job Batch Insert at the end of the Job
487 which might create resource contention with lots of Job,
488 defer the insert to a later time.
490 Why: Permits to focus on getting the data on the Volume and
491 putting the metadata into the Catalog outside the backup
494 Notes: Will use the proposed Bacula ASCII database import/export
495 format (i.e. dependent on the import/export entities project).
498 Item 17: Add MaxVolumeSize/MaxVolumeBytes to Storage resource
499 Origin: Bastian Friedrich <bastian.friedrich@collax.com>
503 What: SD has a "Maximum Volume Size" statement, which is deprecated and
504 superseded by the Pool resource statement "Maximum Volume Bytes".
505 It would be good if either statement could be used in Storage
508 Why: Pools do not have to be restricted to a single storage type/device;
509 thus, it may be impossible to define Maximum Volume Bytes in the
510 Pool resource. The old MaxVolSize statement is deprecated, as it
511 is SD side only. I am using the same pool for different devices.
513 Notes: State of idea currently unknown. Storage resources in the dir
514 config currently translate to very slim catalog entries; these
515 entries would require extensions to implement what is described
516 here. Quite possibly, numerous other statements that are currently
517 available in Pool resources could be used in Storage resources too
521 Item 18: Message mailing based on backup types
522 Origin: Evan Kaufman <evan.kaufman@gmail.com>
523 Date: January 6, 2006
526 What: In the "Messages" resource definitions, allowing messages
527 to be mailed based on the type (backup, restore, etc.) and level
528 (full, differential, etc) of job that created the originating
531 Why: It would, for example, allow someone's boss to be emailed
532 automatically only when a Full Backup job runs, so he can
533 retrieve the tapes for offsite storage, even if the IT dept.
534 doesn't (or can't) explicitly notify him. At the same time, his
535 mailbox wouldnt be filled by notifications of Verifies, Restores,
536 or Incremental/Differential Backups (which would likely be kept
539 Notes: One way this could be done is through additional message types, for
543 # email the boss only on full system backups
544 Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
546 # email us only when something breaks
547 MailOnError = itdept@mycompany.com = all
550 Notes: Kern: This should be rather trivial to implement.
553 Item 19: Handle Windows Encrypted Files using Win raw encryption
554 Origin: Michael Mohr, SAG Mohr.External@infineon.com
555 Date: 22 February 2008
556 Origin: Alex Ehrlich (Alex.Ehrlich-at-mail.ee)
560 What: Make it possible to backup and restore Encypted Files from and to
561 Windows systems without the need to decrypt it by using the raw
562 encryption functions API (see:
563 http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
564 that is provided for that reason by Microsoft.
565 If a file ist encrypted could be examined by evaluating the
566 FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes
568 For each file backed up or restored by FD on Windows, check if
569 the file is encrypted; if so then use OpenEncryptedFileRaw,
570 ReadEncryptedFileRaw, WriteEncryptedFileRaw,
571 CloseEncryptedFileRaw instead of BackupRead and BackupWrite
574 Why: Without the usage of this interface the fd-daemon running
575 under the system account can't read encypted Files because
576 the key needed for the decrytion is missed by them. As a result
577 actually encrypted files are not backed up
578 by bacula and also no error is shown while missing these files.
580 Notes: Using xxxEncryptedFileRaw API would allow to backup and
581 restore EFS-encrypted files without decrypting their data.
582 Note that such files cannot be restored "portably" (at least,
583 easily) but they would be restoreable to a different (or
584 reinstalled) Win32 machine; the restore would require setup
585 of a EFS recovery agent in advance, of course, and this shall
586 be clearly reflected in the documentation, but this is the
587 normal Windows SysAdmin's business.
588 When "portable" backup is requested the EFS-encrypted files
589 shall be clearly reported as errors.
590 See MSDN on the "Backup and Restore of Encrypted Files" topic:
591 http://msdn.microsoft.com/en-us/library/aa363783.aspx
592 Maybe the EFS support requires a new flag in the database for
594 Unfortunately, the implementation is not as straightforward as
595 1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
596 requiring some FD code rewrite to work with
597 encrypted-file-related callback functions.
599 Item 20: Job migration between different SDs
600 Origin: Mariusz Czulada <manieq AT wp DOT eu>
604 What: Allow to specify in migration job devices on Storage Daemon other then
605 the one used for migrated jobs (possibly on different/distant host)
607 Why: Sometimes we have more then one system which requires backup
608 implementation. Often, these systems are functionally unrelated and
609 placed in different locations. Having a big backup device (a tape
610 library) in each location is not cost-effective. It would be much
611 better to have one powerful enough tape library which could handle
612 backups from all systems, assuming relatively fast and reliable WAN
613 connections. In such architecture backups are done in service windows
614 on local bacula servers, then migrated to central storage off the peak
617 Notes: If migration to different SD is working, migration to the same SD, as
618 now, could be done the same way (i mean 'localhost') to unify the
621 Item 19. Allow FD to initiate a backup
622 Origin: Frank Volf (frank at deze dot org)
623 Date: 17 November 2005
626 What: Provide some means, possibly by a restricted console that
627 allows a FD to initiate a backup, and that uses the connection
628 established by the FD to the Director for the backup so that
629 a Director that is firewalled can do the backup.
630 Why: Makes backup of laptops much easier.
631 Notes: - The FD already has code for the monitor interface
632 - It could be nice to have a .job command that lists authorized
634 - Commands need to be restricted on the Director side
635 (for example by re-using the runscript flag)
636 - The Client resource can be used to authorize the connection
637 - In a first time, the client can't modify job parameters
638 - We need a way to run a status command to follow job progression
640 This project consists of the following points
641 1. Modify the FD to have a "mini-console" interface that
642 permits it to connect to the Director and start a
643 backup job of itself.
644 2. The list of jobs that can be started by the FD are
645 defined in the Director (possibly via a restricted
647 3. Modify the existing tray monitor code in the Win32 FD
648 so that it is a separate program from the FD.
649 4. The tray monitor program should be extended to permit
651 5. No new Director directives should be added without
652 prior consultation with the Bacula developers.
653 6. The comm line used by the FD to connect to the Director
654 should be re-used by the Director to do the backup.
655 This feature is partially implemented in the Director.
656 7. The FD may have a new directive that allows it to start
657 a backup when the FD starts.
658 8. The console interface to the FD should be extended to
659 permit a properly authorized console to initiate a
663 Item 21: Implement Storage daemon compression
664 Date: 18 December 2006
665 Origin: Vadim A. Umanski , e-mail umanski@ext.ru
667 What: The ability to compress backup data on the SD receiving data
668 instead of doing that on client sending data.
669 Why: The need is practical. I've got some machines that can send
670 data to the network 4 or 5 times faster than compressing
671 them (I've measured that). They're using fast enough SCSI/FC
672 disk subsystems but rather slow CPUs (ex. UltraSPARC II).
673 And the backup server has got a quite fast CPUs (ex. Dual P4
674 Xeons) and quite a low load. When you have 20, 50 or 100 GB
675 of raw data - running a job 4 to 5 times faster - that
676 really matters. On the other hand, the data can be
677 compressed 50% or better - so losing twice more space for
678 disk backup is not good at all. And the network is all mine
679 (I have a dedicated management/provisioning network) and I
680 can get as high bandwidth as I need - 100Mbps, 1000Mbps...
681 That's why the server-side compression feature is needed!
684 Item 22: Ability to import/export Bacula database entities
689 What: Create a Bacula ASCII SQL database independent format that permits
690 importing and exporting database catalog Job entities.
692 Why: For achival, database clustering, tranfer to other databases
695 Notes: Job selection should be by Job, time, Volume, Client, Pool and possibly
699 Item 23: Implementation of running Job speed limit.
700 Origin: Alex F, alexxzell at yahoo dot com
701 Date: 29 January 2009
703 What: I noticed the need for an integrated bandwidth limiter for
704 running jobs. It would be very useful just to specify another
705 field in bacula-dir.conf, like speed = how much speed you wish
706 for that specific job to run at
708 Why: Because of a couple of reasons. First, it's very hard to implement a
709 traffic shaping utility and also make it reliable. Second, it is very
710 uncomfortable to have to implement these apps to, let's say 50 clients
711 (including desktops, servers). This would also be unreliable because you
712 have to make sure that the apps are properly working when needed; users
713 could also disable them (accidentally or not). It would be very useful
714 to provide Bacula this ability. All information would be centralized,
715 you would not have to go to 50 different clients in 10 different
716 locations for configuration; eliminating 3rd party additions help in
717 establishing efficiency. Would also avoid bandwidth congestion,
718 especially where there is little available.
721 Item 24: Add an override in Schedule for Pools based on backup types
723 Origin: Chad Slater <chad.slater@clickfox.com>
726 What: Adding a FullStorage=BigTapeLibrary in the Schedule resource
727 would help those of us who use different storage devices for different
728 backup levels cope with the "auto-upgrade" of a backup.
730 Why: Assume I add several new devices to be backed up, i.e. several
731 hosts with 1TB RAID. To avoid tape switching hassles, incrementals are
732 stored in a disk set on a 2TB RAID. If you add these devices in the
733 middle of the month, the incrementals are upgraded to "full" backups,
734 but they try to use the same storage device as requested in the
735 incremental job, filling up the RAID holding the differentials. If we
736 could override the Storage parameter for full and/or differential
737 backups, then the Full job would use the proper Storage device, which
738 has more capacity (i.e. a 8TB tape library.
741 Item 25: Automatic promotion of backup levels based on backup size
742 Date: 19 January 2006
743 Origin: Adam Thornton <athornton@sinenomine.net>
746 What: Other backup programs have a feature whereby it estimates the space
747 that a differential, incremental, and full backup would take. If
748 the difference in space required between the scheduled level and the
749 next level up is beneath some user-defined critical threshold, the
750 backup level is bumped to the next type. Doing this minimizes the
751 number of volumes necessary during a restore, with a fairly minimal
752 cost in backup media space.
754 Why: I know at least one (quite sophisticated and smart) user for whom the
755 absence of this feature is a deal-breaker in terms of using Bacula;
756 if we had it it would eliminate the one cool thing other backup
757 programs can do and we can't (at least, the one cool thing I know
761 Item 26: Allow FileSet inclusion/exclusion by creation/mod times
762 Origin: Evan Kaufman <evan.kaufman@gmail.com>
763 Date: January 11, 2006
766 What: In the vein of the Wild and Regex directives in a Fileset's
767 Options, it would be helpful to allow a user to include or exclude
768 files and directories by creation or modification times.
770 You could factor the Exclude=yes|no option in much the same way it
771 affects the Wild and Regex directives. For example, you could exclude
772 all files modified before a certain date:
776 Modified Before = ####
779 Or you could exclude all files created/modified since a certain date:
783 Created Modified Since = ####
786 The format of the time/date could be done several ways, say the number
787 of seconds since the epoch:
788 1137008553 = Jan 11 2006, 1:42:33PM # result of `date +%s`
790 Or a human readable date in a cryptic form:
791 20060111134233 = Jan 11 2006, 1:42:33PM # YYYYMMDDhhmmss
793 Why: I imagine a feature like this could have many uses. It would
794 allow a user to do a full backup while excluding the base operating
795 system files, so if I installed a Linux snapshot from a CD yesterday,
796 I'll *exclude* all files modified *before* today. If I need to
797 recover the system, I use the CD I already have, plus the tape backup.
798 Or if, say, a Windows client is hit by a particularly corrosive
799 virus, and I need to *exclude* any files created/modified *since* the
802 Notes: Of course, this feature would work in concert with other
803 in/exclude rules, and wouldnt override them (or each other).
805 Notes: The directives I'd imagine would be along the lines of
806 "[Created] [Modified] [Before|Since] = <date>".
807 So one could compare against 'ctime' and/or 'mtime', but ONLY 'before'
811 Item 27: Archival (removal) of User Files to Tape
813 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
816 What: The ability to archive data to storage based on certain parameters
817 such as age, size, or location. Once the data has been written to
818 storage and logged it is then pruned from the originating
819 filesystem. Note! We are talking about user's files and not
822 Why: This would allow fully automatic storage management which becomes
823 useful for large datastores. It would also allow for auto-staging
824 from one media type to another.
826 Example 1) Medical imaging needs to store large amounts of data.
827 They decide to keep data on their servers for 6 months and then put
828 it away for long term storage. The server then finds all files
829 older than 6 months writes them to tape. The files are then removed
832 Example 2) All data that hasn't been accessed in 2 months could be
833 moved from high-cost, fibre-channel disk storage to a low-cost
834 large-capacity SATA disk storage pool which doesn't have as quick of
835 access time. Then after another 6 months (or possibly as one
836 storage pool gets full) data is migrated to Tape.
838 Item 28: Ability to reconnect a disconnected comm line
843 What: Often jobs fail because of a communications line drop. In that
844 case, Bacula should be able to reconnect to the other daemon and
847 Why: Avoids backuping data already saved.
849 Notes: *Very* complicated from a design point of view because of authenication.
851 Item 29: Multiple threads in file daemon for the same job
852 Date: 27 November 2005
853 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
856 What: I want the file daemon to start multiple threads for a backup
857 job so the fastest possible backup can be made.
859 The file daemon could parse the FileSet information and start
860 one thread for each File entry located on a separate
863 A confiuration option in the job section should be used to
864 enable or disable this feature. The confgutration option could
865 specify the maximum number of threads in the file daemon.
867 If the theads could spool the data to separate spool files
868 the restore process will not be much slower.
870 Why: Multiple concurrent backups of a large fileserver with many
871 disks and controllers will be much faster.
873 Notes: (KES) This is not necessary and could be accomplished
874 by having two jobs. In addition, the current VSS code
878 Item 30: Automatic disabling of devices
880 Origin: Peter Eriksson <peter at ifm.liu dot se>
883 What: After a configurable amount of fatal errors with a tape drive
884 Bacula should automatically disable further use of a certain
885 tape drive. There should also be "disable"/"enable" commands in
888 Why: On a multi-drive jukebox there is a possibility of tape drives
889 going bad during large backups (needing a cleaning tape run,
890 tapes getting stuck). It would be advantageous if Bacula would
891 automatically disable further use of a problematic tape drive
892 after a configurable amount of errors has occurred.
894 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
895 where tapes occasionally get stuck inside the drive. Bacula will
896 notice that the "mtx-changer" command will fail and then fail
897 any backup jobs trying to use that drive. However, it will still
898 keep on trying to run new jobs using that drive and fail -
899 forever, and thus failing lots and lots of jobs... Since we have
900 many drives Bacula could have just automatically disabled
901 further use of that drive and used one of the other ones
905 Item 31: Enable persistent naming/number of SQL queries
911 Change the parsing of the query.sql file and the query command so that
912 queries are named/numbered by a fixed value, not their order in the
917 One of the real strengths of bacula is the ability to query the
918 database, and the fact that complex queries can be saved and
919 referenced from a file is very powerful. However, the choice
920 of query (both for interactive use, and by scripting input
921 to the bconsole command) is completely dependent on the order
922 within the query.sql file. The descriptve labels are helpful for
923 interactive use, but users become used to calling a particular
924 query "by number", or may use scripts to execute queries. This
925 presents a problem if the number or order of queries in the file
928 If the query.sql file used the numeric tags as a real value (rather
929 than a comment), then users could have a higher confidence that they
930 are executing the intended query, that their local changes wouldn't
931 conflict with future bacula upgrades.
933 For scripting, it's very important that the intended query is
934 what's actually executed. The current method of parsing the
935 query.sql file discourages scripting because the addition or
936 deletion of queries within the file will require corresponding
937 changes to scripts. It may not be obvious to users that deleting
938 query "17" in the query.sql file will require changing all
939 references to higher numbered queries. Similarly, when new
940 bacula distributions change the number of "official" queries,
941 user-developed queries cannot simply be appended to the file
942 without also changing any references to those queries in scripts
943 or procedural documentation, etc.
945 In addition, using fixed numbers for queries would encourage more
946 user-initiated development of queries, by supporting conventions
949 queries numbered 1-50 are supported/developed/distributed by
950 with official bacula releases
952 queries numbered 100-200 are community contributed, and are
953 related to media management
955 queries numbered 201-300 are community contributed, and are
956 related to checksums, finding duplicated files across
957 different backups, etc.
959 queries numbered 301-400 are community contributed, and are
960 related to backup statistics (average file size, size per
961 client per backup level, time for all clients by backup level,
962 storage capacity by media type, etc.)
964 queries numbered 500-999 are locally created
967 Alternatively, queries could be called by keyword (tag), rather
971 Item 32: Bacula Dir, FD and SD to support proxies
972 Origin: Karl Grindley @ MIT Lincoln Laboratory <kgrindley at ll dot mit dot edu>
976 What: Support alternate methods for nailing up a TCP session such
977 as SOCKS5, SOCKS4 and HTTP (CONNECT) proxies. Such a feature
978 would allow tunneling of bacula traffic in and out of proxied
981 Why: Currently, bacula is architected to only function on a flat network, with
982 no barriers or limitations. Due to the large configuration states of
983 any network and the infinite configuration where file daemons and
984 storage daemons may sit in relation to one another, bacula often is
985 not usable on a network where filtered or air-gaped networks exist.
986 While often solutions such as ACL modifications to firewalls or port
987 redirection via SNAT or DNAT will solve the issue, often however,
988 these solutions are not adequate or not allowed by hard policy.
990 In an air-gapped network with only a highly locked down proxy services
991 are provided (SOCKS4/5 and/or HTTP and/or SSH outbound) ACLs or
992 iptable rules will not work.
994 Notes: Director resource tunneling: This configuration option to utilize a
995 proxy to connect to a client should be specified in the client
996 resource Client resource tunneling: should be configured in the client
997 resource in the director config file? Or configured on the bacula-fd
998 configuration file on the fd host itself? If the ladder, this would
999 allow only certain clients to use a proxy, where others do not when
1000 establishing the TCP connection to the storage server.
1002 Also worth noting, there are other 3rd party, light weight apps that
1003 could be utilized to bootstrap this. Instead of sockifing bacula
1004 itself, use an external program to broker proxy authentication, and
1005 connection to the remote host. OpenSSH does this by using the
1006 "ProxyCommand" syntax in the client configuration and uses stdin and
1007 stdout to the command. Connect.c is a very popular one.
1008 (http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html).
1009 One could also possibly use stunnel, netcat, etc.
1012 Item 33: Add Minumum Spool Size directive
1014 Origin: Frank Sweetser <fs@wpi.edu>
1016 What: Add a new SD directive, "minimum spool size" (or similar). This
1017 directive would specify a minimum level of free space available for
1018 spooling. If the unused spool space is less than this level, any
1019 new spooling requests would be blocked as if the "maximum spool
1020 size" threshold had bee reached. Already spooling jobs would be
1021 unaffected by this directive.
1023 Why: I've been bitten by this scenario a couple of times:
1025 Assume a maximum spool size of 100M. Two concurrent jobs, A and B,
1026 are both running. Due to timing quirks and previously running jobs,
1027 job A has used 99.9M of space in the spool directory. While A is
1028 busy despooling to disk, B is happily using the remaining 0.1M of
1029 spool space. This ends up in a spool/despool sequence every 0.1M of
1030 data. In addition to fragmenting the data on the volume far more
1031 than was necessary, in larger data sets (ie, tens or hundreds of
1032 gigabytes) it can easily produce multi-megabyte report emails!
1038 Item 34: Command that releases all drives in an autochanger
1039 Origin: Blake Dunlap (blake@nxs.net)
1043 What: It would be nice if there was a release command that
1044 would release all drives in an autochanger instead of having to
1045 do each one in turn.
1047 Why: It can take some time for a release to occur, and the
1048 commands must be given for each drive in turn, which can quicky
1049 scale if there are several drives in the library. (Having to
1050 watch the console, to give each command can waste a good bit of
1051 time when you start getting into the 16 drive range when the
1052 tapes can take up to 3 minutes to eject each)
1054 Notes: Due to the way some autochangers/libraries work, you
1055 cannot assume that new tapes inserted will go into slots that are
1056 not currently believed to be in use by bacula (the tape from that
1057 slot is in a drive). This would make any changes in
1058 configuration quicker/easier, as all drives need to be released
1059 before any modifications to slots.
1061 Item 35: Run bscan on a remote storage daemon from within bconsole.
1062 Date: 07 October 2009
1063 Origin: Graham Keeling <graham@equiinet.com>
1066 What: The ability to be able to run bscan on a remote storage daemon from
1067 within bconsole in order to populate your catalog.
1069 Why: Currently, it seems you have to:
1070 a) log in to a console on the remote machine
1071 b) figure out where the storage daemon config file is
1072 c) figure out the storage device from the config file
1073 d) figure out the catalog IP address
1074 e) figure out the catalog port
1075 f) open the port on the catalog firewall
1076 g) configure the catalog database to accept connections from the
1078 h) build a 'bscan' command from (b)-(e) above and run it
1079 It would be much nicer to be able to type something like this into
1081 *bscan storage=<storage> device=<device> volume=<volume>
1083 *bscan storage=<storage> all
1084 It seems to me that the scan could also do a better job than the
1085 external bscan program currently does. It would possibly be able to
1086 deduce some extra details, such as the catalog StorageId for the
1089 Notes: (Kern). If you need to do a bscan, you have done something wrong,
1090 so this functionality should not need to be integrated into the
1091 the Storage daemon. However, I am not opposed to someone implementing
1092 this feature providing that all the code is in a shared object (or dll)
1093 and does not add significantly to the size of the Storage daemon. In
1094 addition, the code should be written in a way such that the same source
1095 code is used in both the bscan program and the Storage daemon to avoid
1096 adding a lot of new code that must be maintained by the project.
1098 Item 36: Implement a Migration job type that will create a reverse
1099 incremental (or decremental) backup from two existing full backups.
1100 Date: 05 October 2009
1101 Origin: Griffith College Dublin. Some sponsorship available.
1102 Contact: Gavin McCullagh <gavin.mccullagh@gcd.ie>
1105 What: The ability to take two full backup jobs and derive a reverse
1106 incremental backup from them. The older full backup data may then
1109 Why: Long-term backups based on keeping full backups can be expensive in
1110 media. In many cases (eg a NAS), as the client accumulates files
1111 over months and years, the same file will be duplicated unchanged,
1112 across many media and datasets. Eg, Less than 10% (and
1113 shrinking) of our monthly full mail server backup is new files,
1114 the other 90% is also in the previous full backup.
1115 Regularly converting the oldest full backup into a reverse
1116 incremental backup allows the admin to keep access to old backup
1117 jobs, but remove all of the duplicated files, freeing up media.
1119 Notes: This feature was previously discussed on the bacula-devel list
1120 here: http://www.mail-archive.com/bacula-devel@lists.sourceforge.net/msg04962.html
1122 Item 37: Separate "Storage" and "Device" in the bacula-dir.conf
1124 Origin: "James Harper" <james.harper@bendigoit.com.au>
1125 Status: not implemented or documented
1127 What: Separate "Storage" and "Device" in the bacula-dir.conf
1128 The resulting config would looks something like:
1131 Name = name_of_server
1132 Address = hostname/IP address
1134 Password = shh_its_a_secret
1135 Maximum Concurrent Jobs = 7
1139 Name = name_of_device
1140 Storage = name_of_server
1141 Device = name_of_device_on_sd
1142 Media Type = media_type
1143 Maximum Concurrent Jobs = 1
1146 Maximum Concurrent Jobs would be specified with a server and a device
1147 maximum, which would both be honoured by the director. Almost everything
1148 that mentions a 'Storage' would need to be changed to 'Device', although
1149 perhaps a 'Storage' would just be a synonym for 'Device' for backwards
1152 Why: If you have multiple Storage definitions pointing to different
1153 Devices in the same Storage daemon, the "status storage" command
1154 prompts for each different device, but they all give the same
1159 Item 38: Least recently used device selection for tape drives in autochanger.
1160 Date: 12 October 2009
1161 Origin: Thomas Carter <tcarter@memc.com>
1164 What: A better tape drive selection algorithm for multi-drive
1165 autochangers. The AUTOCHANGER class contains an array list of tape
1166 devices. When a tape drive is needed, this list is always searched in
1167 order. This causes lower number drives (specifically drive 0) to do a
1168 majority of the work with higher numbered drives possibly never being
1169 used. When a drive in an autochanger is reserved for use, its entry should
1170 be moved to the end of the list; this would give a rough LRU drive
1173 Why: The current implementation places a majority of use and wear on drive
1174 0 of a multi-drive autochanger.
1178 Item 39: Implement a Storage device like Amazon's S3.
1179 Date: 25 August 2008
1180 Origin: Soren Hansen <soren@ubuntu.com>
1181 Status: Not started.
1182 What: Enable the storage daemon to store backup data on Amazon's
1185 Why: Amazon's S3 is a cheap way to store data off-site.
1187 Notes: If we configure the Pool to put only one job per volume (they don't
1188 support append operation), and the volume size isn't to big (100MB?),
1189 it should be easy to adapt the disk-changer script to add get/put
1190 procedure with curl. So, the data would be safetly copied during the
1193 Cloud should be only used with Copy jobs, users should always have
1194 a copy of their data on their site.
1196 We should also think to have our own cache, trying always to have
1197 cloud volume on the local disk. (I don't know if users want to store
1198 100GB on cloud, so it shouldn't be a disk size problem). For example,
1199 if bacula want to recycle a volume, it will start by downloading the
1200 file to truncate it few seconds later, if we can avoid that...
1202 Item 40: Convert tray monitor on Windows to a stand alone program
1207 What: Separate Win32 tray monitor to be a separate program.
1209 Why: Vista does not allow SYSTEM services to interact with the
1210 desktop, so the current tray monitor does not work on Vista
1213 Notes: Requires communicating with the FD via the network (simulate
1214 a console connection).
1216 Item 41: Improve Bacula's tape and drive usage and cleaning management
1217 Date: 8 November 2005, November 11, 2005
1218 Origin: Adam Thornton <athornton at sinenomine dot net>,
1219 Arno Lehmann <al at its-lehmann dot de>
1223 1. Measure tape and drive usage (mostly implemented)
1224 2. Retiring a volume when too old or too many errors
1225 3. Handle cleaning and tape alerts.
1230 Item 42: Relabel disk volume after recycling
1231 Origin: Pasi Kärkkäinen <pasik@iki.fi>
1233 Status: Not implemented yet, no code written.
1235 What: The ability to relabel the disk volume (and thus rename the file on the
1236 disk) after it has been recycled. Useful when you have a single job
1237 per disk volume, and you use a custom Label format, for example:
1239 "${Client}-${Level}-${NumVols:p/4/0/r}-${Year}_${Month}_${Day}-${Hour}_${Minute}"
1241 Why: Disk volumes in Bacula get the label/filename when they are used for the
1242 first time. If you use recycling and custom label format like above,
1243 the disk volume name doesn't match the contents after it has been
1244 recycled. This feature makes it possible to keep the label/filename
1245 in sync with the content and thus makes it easy to check/monitor the
1246 backups from the shell and/or normal file management tools, because
1247 the filenames of the disk volumes match the content.
1249 Notes: The configuration option could be "Relabel after Recycling = Yes".
1253 ========= New items after last vote ====================
1256 Note to renumber items use:
1257 scripts/renumber_projects.pl projects >1
1260 ========= Add new items above this line =================
1263 ============= Empty Feature Request form ===========
1264 Item n: One line summary ...
1265 Date: Date submitted
1266 Origin: Name and email of originator.
1269 What: More detailed explanation ...
1271 Why: Why it is important ...
1273 Notes: Additional notes or features (omit if not used)
1274 ============== End Feature Request form ==============
1277 ========== Items put on hold by Kern ============================
1280 ========== Items completed in version 5.0.0 ====================
1281 *Item : 'restore' menu: enter a JobId, automatically select dependents
1282 *Item : Deletion of disk Volumes when pruned (partial -- truncate when pruned)
1283 *Item : Implement Base jobs
1284 *Item : Restore from volumes on multiple storage daemons
1285 *Item : Enable/disable compression depending on storage device (disk/tape)
1286 *Item : Cause daemons to use a specific IP address to source communications
1287 *Item : "Maximum Concurrent Jobs" for drives when used with changer device
1288 *Item : List InChanger flag when doing restore.
1289 *Item : Port bat to Win32
1290 *Item : An option to operate on all pools with update vol parameters
1291 ========== Item completed after 5.0.0 ==========================
1292 *Item : Add ability to Verify any specified Job.