3 Bacula Projects Roadmap
4 Status updated 8 August 2010
9 Item 1: Ability to restart failed jobs
11 Item 3: NDMP backup/restore
12 Item 4: SAP backup/restore
13 Item 5: Oracle backup/restore
14 Item 6: Zimbra and Zarafa backup/restore
15 Item 7: Include timestamp of job launch in "stat clients" output
16 Item 8: Include all conf files in specified directory
17 Item 9: Reduction of communications bandwidth for a backup
18 Item 10: Concurrent spooling and despooling within a single job.
19 Item 11: Start spooling even when waiting on tape
20 Item 12: Add ability to Verify any specified Job.
21 Item 13: Data encryption on storage daemon
22 Item 14: Possibilty to schedule Jobs on last Friday of the month
23 Item 15: Scheduling syntax that permits more flexibility and options
24 Item 16: Ability to defer Batch Insert to a later time
25 Item 17: Add MaxVolumeSize/MaxVolumeBytes to Storage resource
26 Item 18: Message mailing based on backup types
27 Item 19: Handle Windows Encrypted Files using Win raw encryption
28 Item 20: Job migration between different SDs
29 Item 19. Allow FD to initiate a backup
30 Item 21: Implement Storage daemon compression
31 Item 22: Ability to import/export Bacula database entities
32 Item 23: Implementation of running Job speed limit.
33 Item 24: Add an override in Schedule for Pools based on backup types
34 Item 25: Automatic promotion of backup levels based on backup size
35 Item 26: Allow FileSet inclusion/exclusion by creation/mod times
36 Item 27: Archival (removal) of User Files to Tape
37 Item 28: Ability to reconnect a disconnected comm line
38 Item 29: Multiple threads in file daemon for the same job
39 Item 30: Automatic disabling of devices
40 Item 31: Enable persistent naming/number of SQL queries
41 Item 32: Bacula Dir, FD and SD to support proxies
42 Item 33: Add Minumum Spool Size directive
43 Item 34: Command that releases all drives in an autochanger
44 Item 35: Run bscan on a remote storage daemon from within bconsole.
45 Item 36: Implement a Migration job type that will create a reverse
46 Item 37: Extend the verify code to make it possible to verify
47 Item 38: Separate "Storage" and "Device" in the bacula-dir.conf
48 Item 39: Least recently used device selection for tape drives in autochanger.
49 Item 40: Implement a Storage device like Amazon's S3.
50 Item 41: Convert tray monitor on Windows to a stand alone program
51 Item 42: Improve Bacula's tape and drive usage and cleaning management
52 Item 43: Relabel disk volume after recycling
55 Item 1: Ability to restart failed jobs
60 What: Often jobs fail because of a communications line drop or max run time,
61 cancel, or some other non-critical problem. Currrently any data
62 saved is lost. This implementation should modify the Storage daemon
63 so that it saves all the files that it knows are completely backed
66 The jobs should then be marked as incomplete and a subsequent
67 Incremental Accurate backup will then take into account all the
70 Why: Avoids backuping data already saved.
72 Notes: Requires Accurate to restart correctly. Must completed have a minimum
73 volume of data or files stored on Volume before enabling.
80 What: Various ideas for redesigns planned for the SD:
81 1. One thread per drive
82 2. Design a class structure for all objects in the SD.
83 3. Make Device into C++ classes for each device type
84 4. Make Device have a proxy (front end intercept class) that will permit control over locking and changing the real device pointer. It can also permit delaying opening, so that we can adapt to having another program that tells us the Archive device name.
85 5. Allow plugins to create new on the fly devices
86 6. Separate SD volume manager
87 7. Volume manager tells Bacula what drive or device to use for a given volume
89 Why: It will simplify the SD, make it more modular, reduce locking
90 conflicts, and allow multiple buffer backups.
93 Item 3: NDMP backup/restore
95 Origin: Bacula Systems
96 Status: Enterprise only if implemented by Bacula Systems
98 What: Backup/restore via NDMP -- most important NetApp compatibility
102 Item 4: SAP backup/restore
104 Origin: Bacula Systems
105 Status: Enterprise only if implemented by Bacula Systems
107 What: Backup/restore SAP databases (MaxDB, Oracle, possibly DB2)
111 Item 5: Oracle backup/restore
113 Origin: Bacula Systems
114 Status: Enterprise only if implemented by Bacula Systems
116 What: Backup/restore Oracle databases
119 Item 6: Zimbra and Zarafa backup/restore
121 Origin: Bacula Systems
122 Status: Enterprise only if implemented by Bacula Systems
124 What: Backup/restore for Zimbra and Zarafa
128 Item 7: Include timestamp of job launch in "stat clients" output
129 Origin: Mark Bergman <mark.bergman@uphs.upenn.edu>
130 Date: Tue Aug 22 17:13:39 EDT 2006
133 What: The "stat clients" command doesn't include any detail on when
134 the active backup jobs were launched.
136 Why: Including the timestamp would make it much easier to decide whether
137 a job is running properly.
139 Notes: It may be helpful to have the output from "stat clients" formatted
140 more like that from "stat dir" (and other commands), in a column
141 format. The per-client information that's currently shown (level,
142 client name, JobId, Volume, pool, device, Files, etc.) is good, but
143 somewhat hard to parse (both programmatically and visually),
144 particularly when there are many active clients.
147 Item 8: Include all conf files in specified directory
148 Date: 18 October 2008
149 Origin: Database, Lda. Maputo, Mozambique
150 Contact:Cameron Smith / cameron.ord@database.co.mz
153 What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every
154 time Bacula Director restarts or reloads, it will walk the given
155 directory (non-recursively) and include the contents of any files
156 therein, as though they were appended to bacula-dir.conf
158 Why: Permits simplified and safer configuration for larger installations with
159 many client PCs. Currently, through judicious use of JobDefs and
160 similar directives, it is possible to reduce the client-specific part of
161 a configuration to a minimum. The client-specific directives can be
162 prepared according to a standard template and dropped into a known
163 directory. However it is still necessary to add a line to the "master"
164 (bacula-dir.conf) referencing each new file. This exposes the master to
165 unnecessary risk of accidental mistakes and makes automation of adding
166 new client-confs, more difficult (it is easier to automate dropping a
167 file into a dir, than rewriting an existing file). Ken has previously
168 made a convincing argument for NOT including Bacula's core configuration
169 in an RDBMS, but I believe that the present request is a reasonable
170 extension to the current "flat-file-based" configuration philosophy.
172 Notes: There is NO need for any special syntax to these files. They should
173 contain standard directives which are simply "inlined" to the parent
174 file as already happens when you explicitly reference an external file.
176 Notes: (kes) this can already be done with scripting
177 From: John Jorgensen <jorgnsn@lcd.uregina.ca>
178 The bacula-dir.conf at our site contains these lines:
181 # Include subfiles associated with configuration of clients.
182 # They define the bulk of the Clients, Jobs, and FileSets.
184 @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'"
186 and when we get a new client, we just put its configuration into
187 a new file called something like:
189 /etc/bacula/clientdefs/clientname.conf
194 Item 9: Reduction of communications bandwidth for a backup
195 Date: 14 October 2008
196 Origin: Robin O'Leary (Equiinet)
199 What: Using rdiff techniques, Bacula could significantly reduce
200 the network data transfer volume to do a backup.
202 Why: Faster backup across the Internet
204 Notes: This requires retaining certain data on the client during a Full
205 backup that will speed up subsequent backups.
208 Item 10: Concurrent spooling and despooling within a single job.
210 Origin: Jesper Krogh <jesper@krogh.cc>
212 What: When a job has spooling enabled and the spool area size is
213 less than the total volumes size the storage daemon will:
214 1) Spool to spool-area
216 3) Go to 1 if more data to be backed up.
218 Typical disks will serve data with a speed of 100MB/s when
219 dealing with large files, network it typical capable of doing 115MB/s
220 (GbitE). Tape drives will despool with 50-90MB/s (LTO3) 70-120MB/s
221 (LTO4) depending on compression and data.
223 As bacula currently works it'll hold back data from the client until
224 de-spooling is done, now matter if the spool area can handle another
225 block of data. Say given a FileSet of 4TB and a spool-area of 100GB and
226 a Maximum Job Spool Size set to 50GB then above sequence could be
227 changed to allow to spool to the other 50GB while despooling the first
228 50GB and not holding back the client while doing it. As above numbers
229 show, depending on tape-drive and disk-arrays this potentially leads to
230 a cut of the backup-time of 50% for the individual jobs.
232 Real-world example, backing up 112.6GB (large files) to LTO4 tapes
233 (despools with ~75MB/s, data is gzipped on the remote filesystem.
234 Maximum Job Spool Size = 8GB
238 Elapsed time (total time): 46m 15s => 2775s
239 Despooling time: 25m 41s => 1541s (55%)
240 Spooling time: 20m 34s => 1234s (45%)
241 Reported speed: 40.58MB/s
242 Spooling speed: 112.6GB/1234s => 91.25MB/s
243 Despooling speed: 112.6GB/1541s => 73.07MB/s
245 So disk + net can "keep up" with the LTO4 drive (in this test)
247 Prosed change would effectively make the backup run in the "despooling
248 time" 1541s giving a reduction to 55% of the total run time.
250 In the situation where the individual job cannot keep up with LTO-drive
251 spooling enables efficient multiplexing of multiple concurrent jobs onto
254 Why: When dealing with larger volumes the general utillization of the
255 network/disk is important to maximize in order to be able to run a full
256 backup over a weekend. Current work-around is to split the FileSet in
257 smaller FileSet and Jobs but that leads to more configuration mangement
258 and is harder to review for completeness. Subsequently it makes restores
263 Item 11: Start spooling even when waiting on tape
264 Origin: Tobias Barth <tobias.barth@web-arts.com>
268 What: If a job can be spooled to disk before writing it to tape, it should
269 be spooled immediately. Currently, bacula waits until the correct
270 tape is inserted into the drive.
272 Why: It could save hours. When bacula waits on the operator who must insert
273 the correct tape (e.g. a new tape or a tape from another media
274 pool), bacula could already prepare the spooled data in the spooling
275 directory and immediately start despooling when the tape was
276 inserted by the operator.
278 2nd step: Use 2 or more spooling directories. When one directory is
279 currently despooling, the next (on different disk drives) could
280 already be spooling the next data.
282 Notes: I am using bacula 2.2.8, which has none of those features
286 Item 12: Add ability to Verify any specified Job.
287 Date: 17 January 2008
288 Origin: portrix.net Hamburg, Germany.
289 Contact: Christian Sabelmann
290 Status: 70% of the required Code is part of the Verify function since v. 2.x
293 The ability to tell Bacula which Job should verify instead of
294 automatically verify just the last one.
297 It is sad that such a powerfull feature like Verify Jobs
298 (VolumeToCatalog) is restricted to be used only with the last backup Job
299 of a client. Actual users who have to do daily Backups are forced to
300 also do daily Verify Jobs in order to take advantage of this useful
301 feature. This Daily Verify after Backup conduct is not always desired
302 and Verify Jobs have to be sometimes scheduled. (Not necessarily
303 scheduled in Bacula). With this feature Admins can verify Jobs once a
304 Week or less per month, selecting the Jobs they want to verify. This
305 feature is also not to difficult to implement taking in account older bug
306 reports about this feature and the selection of the Job to be verified.
308 Notes: For the verify Job, the user could select the Job to be verified
309 from a List of the latest Jobs of a client. It would also be possible to
310 verify a certain volume. All of these would naturaly apply only for
311 Jobs whose file information are still in the catalog.
314 Item 13: Data encryption on storage daemon
315 Origin: Tobias Barth <tobias.barth at web-arts.com>
316 Date: 04 February 2009
319 What: The storage demon should be able to do the data encryption that can
320 currently be done by the file daemon.
322 Why: This would have 2 advantages:
323 1) one could encrypt the data of unencrypted tapes by doing a
325 2) the storage daemon would be the only machine that would have
326 to keep the encryption keys.
329 As an addendum to the feature request, here are some crypto
330 implementation details I wrote up regarding SD-encryption back in Jan
332 http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html
336 Item 14: Possibilty to schedule Jobs on last Friday of the month
337 Origin: Carsten Menke <bootsy52 at gmx dot net>
341 What: Currently if you want to run your monthly Backups on the last
342 Friday of each month this is only possible with workarounds (e.g
343 scripting) (As some months got 4 Fridays and some got 5 Fridays)
344 The same is true if you plan to run your yearly Backups on the
345 last Friday of the year. It would be nice to have the ability to
346 use the builtin scheduler for this.
348 Why: In many companies the last working day of the week is Friday (or
349 Saturday), so to get the most data of the month onto the monthly
350 tape, the employees are advised to insert the tape for the
351 monthly backups on the last friday of the month.
353 Notes: To give this a complete functionality it would be nice if the
354 "first" and "last" Keywords could be implemented in the
355 scheduler, so it is also possible to run monthy backups at the
356 first friday of the month and many things more. So if the syntax
357 would expand to this {first|last} {Month|Week|Day|Mo-Fri} of the
358 {Year|Month|Week} you would be able to run really flexible jobs.
360 To got a certain Job run on the last Friday of the Month for example
363 Run = pool=Monthly last Fri of the Month at 23:50
367 Run = pool=Yearly last Fri of the Year at 23:50
369 ## Certain Jobs the last Week of a Month
371 Run = pool=LastWeek last Week of the Month at 23:50
373 ## Monthly Backup on the last day of the month
375 Run = pool=Monthly last Day of the Month at 23:50
377 Item 15: Scheduling syntax that permits more flexibility and options
378 Date: 15 December 2006
379 Origin: Gregory Brauer (greg at wildbrain dot com) and
380 Florian Schnabel <florian.schnabel at docufy dot de>
383 What: Currently, Bacula only understands how to deal with weeks of the
384 month or weeks of the year in schedules. This makes it impossible
385 to do a true weekly rotation of tapes. There will always be a
386 discontinuity that will require disruptive manual intervention at
387 least monthly or yearly because week boundaries never align with
388 month or year boundaries.
390 A solution would be to add a new syntax that defines (at least)
391 a start timestamp, and repetition period.
393 An easy option to skip a certain job on a certain date.
396 Why: Rotated backups done at weekly intervals are useful, and Bacula
397 cannot currently do them without extensive hacking.
399 You could then easily skip tape backups on holidays. Especially
400 if you got no autochanger and can only fit one backup on a tape
401 that would be really handy, other jobs could proceed normally
402 and you won't get errors that way.
405 Notes: Here is an example syntax showing a 3-week rotation where full
406 Backups would be performed every week on Saturday, and an
407 incremental would be performed every week on Tuesday. Each
408 set of tapes could be removed from the loader for the following
409 two cycles before coming back and being reused on the third
410 week. Since the execution times are determined by intervals
411 from a given point in time, there will never be any issues with
412 having to adjust to any sort of arbitrary time boundary. In
413 the example provided, I even define the starting schedule
414 as crossing both a year and a month boundary, but the run times
415 would be based on the "Repeat" value and would therefore happen
420 Name = "Week 1 Rotation"
421 #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
425 Start = 2006-12-30 01:00
429 #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
433 Start = 2007-01-02 01:00
440 Name = "Week 2 Rotation"
441 #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
445 Start = 2007-01-06 01:00
449 #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
453 Start = 2007-01-09 01:00
460 Name = "Week 3 Rotation"
461 #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
465 Start = 2007-01-13 01:00
469 #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
473 Start = 2007-01-16 01:00
479 Notes: Kern: I have merged the previously separate project of skipping
480 jobs (via Schedule syntax) into this.
483 Item 16: Ability to defer Batch Insert to a later time
488 What: Instead of doing a Job Batch Insert at the end of the Job
489 which might create resource contention with lots of Job,
490 defer the insert to a later time.
492 Why: Permits to focus on getting the data on the Volume and
493 putting the metadata into the Catalog outside the backup
496 Notes: Will use the proposed Bacula ASCII database import/export
497 format (i.e. dependent on the import/export entities project).
500 Item 17: Add MaxVolumeSize/MaxVolumeBytes to Storage resource
501 Origin: Bastian Friedrich <bastian.friedrich@collax.com>
505 What: SD has a "Maximum Volume Size" statement, which is deprecated and
506 superseded by the Pool resource statement "Maximum Volume Bytes".
507 It would be good if either statement could be used in Storage
510 Why: Pools do not have to be restricted to a single storage type/device;
511 thus, it may be impossible to define Maximum Volume Bytes in the
512 Pool resource. The old MaxVolSize statement is deprecated, as it
513 is SD side only. I am using the same pool for different devices.
515 Notes: State of idea currently unknown. Storage resources in the dir
516 config currently translate to very slim catalog entries; these
517 entries would require extensions to implement what is described
518 here. Quite possibly, numerous other statements that are currently
519 available in Pool resources could be used in Storage resources too
523 Item 18: Message mailing based on backup types
524 Origin: Evan Kaufman <evan.kaufman@gmail.com>
525 Date: January 6, 2006
528 What: In the "Messages" resource definitions, allowing messages
529 to be mailed based on the type (backup, restore, etc.) and level
530 (full, differential, etc) of job that created the originating
533 Why: It would, for example, allow someone's boss to be emailed
534 automatically only when a Full Backup job runs, so he can
535 retrieve the tapes for offsite storage, even if the IT dept.
536 doesn't (or can't) explicitly notify him. At the same time, his
537 mailbox wouldnt be filled by notifications of Verifies, Restores,
538 or Incremental/Differential Backups (which would likely be kept
541 Notes: One way this could be done is through additional message types, for
545 # email the boss only on full system backups
546 Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
548 # email us only when something breaks
549 MailOnError = itdept@mycompany.com = all
552 Notes: Kern: This should be rather trivial to implement.
555 Item 19: Handle Windows Encrypted Files using Win raw encryption
556 Origin: Michael Mohr, SAG Mohr.External@infineon.com
557 Date: 22 February 2008
558 Origin: Alex Ehrlich (Alex.Ehrlich-at-mail.ee)
562 What: Make it possible to backup and restore Encypted Files from and to
563 Windows systems without the need to decrypt it by using the raw
564 encryption functions API (see:
565 http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
566 that is provided for that reason by Microsoft.
567 If a file ist encrypted could be examined by evaluating the
568 FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes
570 For each file backed up or restored by FD on Windows, check if
571 the file is encrypted; if so then use OpenEncryptedFileRaw,
572 ReadEncryptedFileRaw, WriteEncryptedFileRaw,
573 CloseEncryptedFileRaw instead of BackupRead and BackupWrite
576 Why: Without the usage of this interface the fd-daemon running
577 under the system account can't read encypted Files because
578 the key needed for the decrytion is missed by them. As a result
579 actually encrypted files are not backed up
580 by bacula and also no error is shown while missing these files.
582 Notes: Using xxxEncryptedFileRaw API would allow to backup and
583 restore EFS-encrypted files without decrypting their data.
584 Note that such files cannot be restored "portably" (at least,
585 easily) but they would be restoreable to a different (or
586 reinstalled) Win32 machine; the restore would require setup
587 of a EFS recovery agent in advance, of course, and this shall
588 be clearly reflected in the documentation, but this is the
589 normal Windows SysAdmin's business.
590 When "portable" backup is requested the EFS-encrypted files
591 shall be clearly reported as errors.
592 See MSDN on the "Backup and Restore of Encrypted Files" topic:
593 http://msdn.microsoft.com/en-us/library/aa363783.aspx
594 Maybe the EFS support requires a new flag in the database for
596 Unfortunately, the implementation is not as straightforward as
597 1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
598 requiring some FD code rewrite to work with
599 encrypted-file-related callback functions.
601 Item 20: Job migration between different SDs
602 Origin: Mariusz Czulada <manieq AT wp DOT eu>
606 What: Allow to specify in migration job devices on Storage Daemon other then
607 the one used for migrated jobs (possibly on different/distant host)
609 Why: Sometimes we have more then one system which requires backup
610 implementation. Often, these systems are functionally unrelated and
611 placed in different locations. Having a big backup device (a tape
612 library) in each location is not cost-effective. It would be much
613 better to have one powerful enough tape library which could handle
614 backups from all systems, assuming relatively fast and reliable WAN
615 connections. In such architecture backups are done in service windows
616 on local bacula servers, then migrated to central storage off the peak
619 Notes: If migration to different SD is working, migration to the same SD, as
620 now, could be done the same way (i mean 'localhost') to unify the
623 Item 19. Allow FD to initiate a backup
624 Origin: Frank Volf (frank at deze dot org)
625 Date: 17 November 2005
628 What: Provide some means, possibly by a restricted console that
629 allows a FD to initiate a backup, and that uses the connection
630 established by the FD to the Director for the backup so that
631 a Director that is firewalled can do the backup.
632 Why: Makes backup of laptops much easier.
633 Notes: - The FD already has code for the monitor interface
634 - It could be nice to have a .job command that lists authorized
636 - Commands need to be restricted on the Director side
637 (for example by re-using the runscript flag)
638 - The Client resource can be used to authorize the connection
639 - In a first time, the client can't modify job parameters
640 - We need a way to run a status command to follow job progression
642 This project consists of the following points
643 1. Modify the FD to have a "mini-console" interface that
644 permits it to connect to the Director and start a
645 backup job of itself.
646 2. The list of jobs that can be started by the FD are
647 defined in the Director (possibly via a restricted
649 3. Modify the existing tray monitor code in the Win32 FD
650 so that it is a separate program from the FD.
651 4. The tray monitor program should be extended to permit
653 5. No new Director directives should be added without
654 prior consultation with the Bacula developers.
655 6. The comm line used by the FD to connect to the Director
656 should be re-used by the Director to do the backup.
657 This feature is partially implemented in the Director.
658 7. The FD may have a new directive that allows it to start
659 a backup when the FD starts.
660 8. The console interface to the FD should be extended to
661 permit a properly authorized console to initiate a
665 Item 21: Implement Storage daemon compression
666 Date: 18 December 2006
667 Origin: Vadim A. Umanski , e-mail umanski@ext.ru
669 What: The ability to compress backup data on the SD receiving data
670 instead of doing that on client sending data.
671 Why: The need is practical. I've got some machines that can send
672 data to the network 4 or 5 times faster than compressing
673 them (I've measured that). They're using fast enough SCSI/FC
674 disk subsystems but rather slow CPUs (ex. UltraSPARC II).
675 And the backup server has got a quite fast CPUs (ex. Dual P4
676 Xeons) and quite a low load. When you have 20, 50 or 100 GB
677 of raw data - running a job 4 to 5 times faster - that
678 really matters. On the other hand, the data can be
679 compressed 50% or better - so losing twice more space for
680 disk backup is not good at all. And the network is all mine
681 (I have a dedicated management/provisioning network) and I
682 can get as high bandwidth as I need - 100Mbps, 1000Mbps...
683 That's why the server-side compression feature is needed!
686 Item 22: Ability to import/export Bacula database entities
691 What: Create a Bacula ASCII SQL database independent format that permits
692 importing and exporting database catalog Job entities.
694 Why: For achival, database clustering, tranfer to other databases
697 Notes: Job selection should be by Job, time, Volume, Client, Pool and possibly
701 Item 23: Implementation of running Job speed limit.
702 Origin: Alex F, alexxzell at yahoo dot com
703 Date: 29 January 2009
705 What: I noticed the need for an integrated bandwidth limiter for
706 running jobs. It would be very useful just to specify another
707 field in bacula-dir.conf, like speed = how much speed you wish
708 for that specific job to run at
710 Why: Because of a couple of reasons. First, it's very hard to implement a
711 traffic shaping utility and also make it reliable. Second, it is very
712 uncomfortable to have to implement these apps to, let's say 50 clients
713 (including desktops, servers). This would also be unreliable because you
714 have to make sure that the apps are properly working when needed; users
715 could also disable them (accidentally or not). It would be very useful
716 to provide Bacula this ability. All information would be centralized,
717 you would not have to go to 50 different clients in 10 different
718 locations for configuration; eliminating 3rd party additions help in
719 establishing efficiency. Would also avoid bandwidth congestion,
720 especially where there is little available.
723 Item 24: Add an override in Schedule for Pools based on backup types
725 Origin: Chad Slater <chad.slater@clickfox.com>
728 What: Adding a FullStorage=BigTapeLibrary in the Schedule resource
729 would help those of us who use different storage devices for different
730 backup levels cope with the "auto-upgrade" of a backup.
732 Why: Assume I add several new devices to be backed up, i.e. several
733 hosts with 1TB RAID. To avoid tape switching hassles, incrementals are
734 stored in a disk set on a 2TB RAID. If you add these devices in the
735 middle of the month, the incrementals are upgraded to "full" backups,
736 but they try to use the same storage device as requested in the
737 incremental job, filling up the RAID holding the differentials. If we
738 could override the Storage parameter for full and/or differential
739 backups, then the Full job would use the proper Storage device, which
740 has more capacity (i.e. a 8TB tape library.
743 Item 25: Automatic promotion of backup levels based on backup size
744 Date: 19 January 2006
745 Origin: Adam Thornton <athornton@sinenomine.net>
748 What: Other backup programs have a feature whereby it estimates the space
749 that a differential, incremental, and full backup would take. If
750 the difference in space required between the scheduled level and the
751 next level up is beneath some user-defined critical threshold, the
752 backup level is bumped to the next type. Doing this minimizes the
753 number of volumes necessary during a restore, with a fairly minimal
754 cost in backup media space.
756 Why: I know at least one (quite sophisticated and smart) user for whom the
757 absence of this feature is a deal-breaker in terms of using Bacula;
758 if we had it it would eliminate the one cool thing other backup
759 programs can do and we can't (at least, the one cool thing I know
763 Item 26: Allow FileSet inclusion/exclusion by creation/mod times
764 Origin: Evan Kaufman <evan.kaufman@gmail.com>
765 Date: January 11, 2006
768 What: In the vein of the Wild and Regex directives in a Fileset's
769 Options, it would be helpful to allow a user to include or exclude
770 files and directories by creation or modification times.
772 You could factor the Exclude=yes|no option in much the same way it
773 affects the Wild and Regex directives. For example, you could exclude
774 all files modified before a certain date:
778 Modified Before = ####
781 Or you could exclude all files created/modified since a certain date:
785 Created Modified Since = ####
788 The format of the time/date could be done several ways, say the number
789 of seconds since the epoch:
790 1137008553 = Jan 11 2006, 1:42:33PM # result of `date +%s`
792 Or a human readable date in a cryptic form:
793 20060111134233 = Jan 11 2006, 1:42:33PM # YYYYMMDDhhmmss
795 Why: I imagine a feature like this could have many uses. It would
796 allow a user to do a full backup while excluding the base operating
797 system files, so if I installed a Linux snapshot from a CD yesterday,
798 I'll *exclude* all files modified *before* today. If I need to
799 recover the system, I use the CD I already have, plus the tape backup.
800 Or if, say, a Windows client is hit by a particularly corrosive
801 virus, and I need to *exclude* any files created/modified *since* the
804 Notes: Of course, this feature would work in concert with other
805 in/exclude rules, and wouldnt override them (or each other).
807 Notes: The directives I'd imagine would be along the lines of
808 "[Created] [Modified] [Before|Since] = <date>".
809 So one could compare against 'ctime' and/or 'mtime', but ONLY 'before'
813 Item 27: Archival (removal) of User Files to Tape
815 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
818 What: The ability to archive data to storage based on certain parameters
819 such as age, size, or location. Once the data has been written to
820 storage and logged it is then pruned from the originating
821 filesystem. Note! We are talking about user's files and not
824 Why: This would allow fully automatic storage management which becomes
825 useful for large datastores. It would also allow for auto-staging
826 from one media type to another.
828 Example 1) Medical imaging needs to store large amounts of data.
829 They decide to keep data on their servers for 6 months and then put
830 it away for long term storage. The server then finds all files
831 older than 6 months writes them to tape. The files are then removed
834 Example 2) All data that hasn't been accessed in 2 months could be
835 moved from high-cost, fibre-channel disk storage to a low-cost
836 large-capacity SATA disk storage pool which doesn't have as quick of
837 access time. Then after another 6 months (or possibly as one
838 storage pool gets full) data is migrated to Tape.
840 Item 28: Ability to reconnect a disconnected comm line
845 What: Often jobs fail because of a communications line drop. In that
846 case, Bacula should be able to reconnect to the other daemon and
849 Why: Avoids backuping data already saved.
851 Notes: *Very* complicated from a design point of view because of authenication.
853 Item 29: Multiple threads in file daemon for the same job
854 Date: 27 November 2005
855 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
858 What: I want the file daemon to start multiple threads for a backup
859 job so the fastest possible backup can be made.
861 The file daemon could parse the FileSet information and start
862 one thread for each File entry located on a separate
865 A confiuration option in the job section should be used to
866 enable or disable this feature. The confgutration option could
867 specify the maximum number of threads in the file daemon.
869 If the theads could spool the data to separate spool files
870 the restore process will not be much slower.
872 Why: Multiple concurrent backups of a large fileserver with many
873 disks and controllers will be much faster.
875 Notes: (KES) This is not necessary and could be accomplished
876 by having two jobs. In addition, the current VSS code
880 Item 30: Automatic disabling of devices
882 Origin: Peter Eriksson <peter at ifm.liu dot se>
885 What: After a configurable amount of fatal errors with a tape drive
886 Bacula should automatically disable further use of a certain
887 tape drive. There should also be "disable"/"enable" commands in
890 Why: On a multi-drive jukebox there is a possibility of tape drives
891 going bad during large backups (needing a cleaning tape run,
892 tapes getting stuck). It would be advantageous if Bacula would
893 automatically disable further use of a problematic tape drive
894 after a configurable amount of errors has occurred.
896 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
897 where tapes occasionally get stuck inside the drive. Bacula will
898 notice that the "mtx-changer" command will fail and then fail
899 any backup jobs trying to use that drive. However, it will still
900 keep on trying to run new jobs using that drive and fail -
901 forever, and thus failing lots and lots of jobs... Since we have
902 many drives Bacula could have just automatically disabled
903 further use of that drive and used one of the other ones
907 Item 31: Enable persistent naming/number of SQL queries
913 Change the parsing of the query.sql file and the query command so that
914 queries are named/numbered by a fixed value, not their order in the
919 One of the real strengths of bacula is the ability to query the
920 database, and the fact that complex queries can be saved and
921 referenced from a file is very powerful. However, the choice
922 of query (both for interactive use, and by scripting input
923 to the bconsole command) is completely dependent on the order
924 within the query.sql file. The descriptve labels are helpful for
925 interactive use, but users become used to calling a particular
926 query "by number", or may use scripts to execute queries. This
927 presents a problem if the number or order of queries in the file
930 If the query.sql file used the numeric tags as a real value (rather
931 than a comment), then users could have a higher confidence that they
932 are executing the intended query, that their local changes wouldn't
933 conflict with future bacula upgrades.
935 For scripting, it's very important that the intended query is
936 what's actually executed. The current method of parsing the
937 query.sql file discourages scripting because the addition or
938 deletion of queries within the file will require corresponding
939 changes to scripts. It may not be obvious to users that deleting
940 query "17" in the query.sql file will require changing all
941 references to higher numbered queries. Similarly, when new
942 bacula distributions change the number of "official" queries,
943 user-developed queries cannot simply be appended to the file
944 without also changing any references to those queries in scripts
945 or procedural documentation, etc.
947 In addition, using fixed numbers for queries would encourage more
948 user-initiated development of queries, by supporting conventions
951 queries numbered 1-50 are supported/developed/distributed by
952 with official bacula releases
954 queries numbered 100-200 are community contributed, and are
955 related to media management
957 queries numbered 201-300 are community contributed, and are
958 related to checksums, finding duplicated files across
959 different backups, etc.
961 queries numbered 301-400 are community contributed, and are
962 related to backup statistics (average file size, size per
963 client per backup level, time for all clients by backup level,
964 storage capacity by media type, etc.)
966 queries numbered 500-999 are locally created
969 Alternatively, queries could be called by keyword (tag), rather
973 Item 32: Bacula Dir, FD and SD to support proxies
974 Origin: Karl Grindley @ MIT Lincoln Laboratory <kgrindley at ll dot mit dot edu>
978 What: Support alternate methods for nailing up a TCP session such
979 as SOCKS5, SOCKS4 and HTTP (CONNECT) proxies. Such a feature
980 would allow tunneling of bacula traffic in and out of proxied
983 Why: Currently, bacula is architected to only function on a flat network, with
984 no barriers or limitations. Due to the large configuration states of
985 any network and the infinite configuration where file daemons and
986 storage daemons may sit in relation to one another, bacula often is
987 not usable on a network where filtered or air-gaped networks exist.
988 While often solutions such as ACL modifications to firewalls or port
989 redirection via SNAT or DNAT will solve the issue, often however,
990 these solutions are not adequate or not allowed by hard policy.
992 In an air-gapped network with only a highly locked down proxy services
993 are provided (SOCKS4/5 and/or HTTP and/or SSH outbound) ACLs or
994 iptable rules will not work.
996 Notes: Director resource tunneling: This configuration option to utilize a
997 proxy to connect to a client should be specified in the client
998 resource Client resource tunneling: should be configured in the client
999 resource in the director config file? Or configured on the bacula-fd
1000 configuration file on the fd host itself? If the ladder, this would
1001 allow only certain clients to use a proxy, where others do not when
1002 establishing the TCP connection to the storage server.
1004 Also worth noting, there are other 3rd party, light weight apps that
1005 could be utilized to bootstrap this. Instead of sockifing bacula
1006 itself, use an external program to broker proxy authentication, and
1007 connection to the remote host. OpenSSH does this by using the
1008 "ProxyCommand" syntax in the client configuration and uses stdin and
1009 stdout to the command. Connect.c is a very popular one.
1010 (http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html).
1011 One could also possibly use stunnel, netcat, etc.
1014 Item 33: Add Minumum Spool Size directive
1016 Origin: Frank Sweetser <fs@wpi.edu>
1018 What: Add a new SD directive, "minimum spool size" (or similar). This
1019 directive would specify a minimum level of free space available for
1020 spooling. If the unused spool space is less than this level, any
1021 new spooling requests would be blocked as if the "maximum spool
1022 size" threshold had bee reached. Already spooling jobs would be
1023 unaffected by this directive.
1025 Why: I've been bitten by this scenario a couple of times:
1027 Assume a maximum spool size of 100M. Two concurrent jobs, A and B,
1028 are both running. Due to timing quirks and previously running jobs,
1029 job A has used 99.9M of space in the spool directory. While A is
1030 busy despooling to disk, B is happily using the remaining 0.1M of
1031 spool space. This ends up in a spool/despool sequence every 0.1M of
1032 data. In addition to fragmenting the data on the volume far more
1033 than was necessary, in larger data sets (ie, tens or hundreds of
1034 gigabytes) it can easily produce multi-megabyte report emails!
1040 Item 34: Command that releases all drives in an autochanger
1041 Origin: Blake Dunlap (blake@nxs.net)
1045 What: It would be nice if there was a release command that
1046 would release all drives in an autochanger instead of having to
1047 do each one in turn.
1049 Why: It can take some time for a release to occur, and the
1050 commands must be given for each drive in turn, which can quicky
1051 scale if there are several drives in the library. (Having to
1052 watch the console, to give each command can waste a good bit of
1053 time when you start getting into the 16 drive range when the
1054 tapes can take up to 3 minutes to eject each)
1056 Notes: Due to the way some autochangers/libraries work, you
1057 cannot assume that new tapes inserted will go into slots that are
1058 not currently believed to be in use by bacula (the tape from that
1059 slot is in a drive). This would make any changes in
1060 configuration quicker/easier, as all drives need to be released
1061 before any modifications to slots.
1063 Item 35: Run bscan on a remote storage daemon from within bconsole.
1064 Date: 07 October 2009
1065 Origin: Graham Keeling <graham@equiinet.com>
1068 What: The ability to be able to run bscan on a remote storage daemon from
1069 within bconsole in order to populate your catalog.
1071 Why: Currently, it seems you have to:
1072 a) log in to a console on the remote machine
1073 b) figure out where the storage daemon config file is
1074 c) figure out the storage device from the config file
1075 d) figure out the catalog IP address
1076 e) figure out the catalog port
1077 f) open the port on the catalog firewall
1078 g) configure the catalog database to accept connections from the
1080 h) build a 'bscan' command from (b)-(e) above and run it
1081 It would be much nicer to be able to type something like this into
1083 *bscan storage=<storage> device=<device> volume=<volume>
1085 *bscan storage=<storage> all
1086 It seems to me that the scan could also do a better job than the
1087 external bscan program currently does. It would possibly be able to
1088 deduce some extra details, such as the catalog StorageId for the
1091 Notes: (Kern). If you need to do a bscan, you have done something wrong,
1092 so this functionality should not need to be integrated into the
1093 the Storage daemon. However, I am not opposed to someone implementing
1094 this feature providing that all the code is in a shared object (or dll)
1095 and does not add significantly to the size of the Storage daemon. In
1096 addition, the code should be written in a way such that the same source
1097 code is used in both the bscan program and the Storage daemon to avoid
1098 adding a lot of new code that must be maintained by the project.
1100 Item 36: Implement a Migration job type that will create a reverse
1101 incremental (or decremental) backup from two existing full backups.
1102 Date: 05 October 2009
1103 Origin: Griffith College Dublin. Some sponsorship available.
1104 Contact: Gavin McCullagh <gavin.mccullagh@gcd.ie>
1107 What: The ability to take two full backup jobs and derive a reverse
1108 incremental backup from them. The older full backup data may then
1111 Why: Long-term backups based on keeping full backups can be expensive in
1112 media. In many cases (eg a NAS), as the client accumulates files
1113 over months and years, the same file will be duplicated unchanged,
1114 across many media and datasets. Eg, Less than 10% (and
1115 shrinking) of our monthly full mail server backup is new files,
1116 the other 90% is also in the previous full backup.
1117 Regularly converting the oldest full backup into a reverse
1118 incremental backup allows the admin to keep access to old backup
1119 jobs, but remove all of the duplicated files, freeing up media.
1121 Notes: This feature was previously discussed on the bacula-devel list
1122 here: http://www.mail-archive.com/bacula-devel@lists.sourceforge.net/msg04962.html
1126 Item 37: Extend the verify code to make it possible to verify
1127 older jobs, not only the last one that has finished
1129 Origin: Ralf Gross (Ralf-Lists <at> ralfgross.de)
1130 Status: not implemented or documented
1132 What: At the moment a VolumeToCatalog job compares only the
1133 last job with the data in the catalog. It's not possible
1134 to compare the data (md5sums) of an older volume with the
1135 data in the catalog.
1137 Why: If a verify job fails, one has to immediately check the
1138 source of the problem, fix it and rerun the verify job.
1139 This has to happen before the next backup of the
1140 verified backup job starts.
1141 More important: It's not possible to check jobs that are
1142 kept for a long time (archiv). If a jobid could be
1143 specified for a verify job, older backups/tapes could be
1144 checked on a regular base.
1146 Notes: verify documentation:
1147 VolumeToCatalog: This level causes Bacula to read the file
1148 attribute data written to the Volume from the last Job [...]
1150 Verify Job = <Job-Resource-Name> If you run a verify job
1151 without this directive, the last job run will be compared
1152 with the catalog, which means that you must immediately
1153 follow a backup by a verify command. If you specify a Verify
1154 Job Bacula will find the last job with that name that ran [...]
1156 example bconsole verify dialog:
1159 JobName: VerifyServerXXX
1160 Level: VolumeToCatalog
1161 Client: ServerXXX-fd
1162 FileSet: ServerXXX-Vol1
1163 Pool: Full (From Job resource)
1164 Storage: Neo4100 (From Pool resource)
1165 Verify Job: ServerXXX-Vol1
1167 When: 2009-04-20 09:03:04
1169 OK to run? (yes/mod/no): m
1170 Parameters to modify:
1183 Item 38: Separate "Storage" and "Device" in the bacula-dir.conf
1185 Origin: "James Harper" <james.harper@bendigoit.com.au>
1186 Status: not implemented or documented
1188 What: Separate "Storage" and "Device" in the bacula-dir.conf
1189 The resulting config would looks something like:
1192 Name = name_of_server
1193 Address = hostname/IP address
1195 Password = shh_its_a_secret
1196 Maximum Concurrent Jobs = 7
1200 Name = name_of_device
1201 Storage = name_of_server
1202 Device = name_of_device_on_sd
1203 Media Type = media_type
1204 Maximum Concurrent Jobs = 1
1207 Maximum Concurrent Jobs would be specified with a server and a device
1208 maximum, which would both be honoured by the director. Almost everything
1209 that mentions a 'Storage' would need to be changed to 'Device', although
1210 perhaps a 'Storage' would just be a synonym for 'Device' for backwards
1213 Why: If you have multiple Storage definitions pointing to different
1214 Devices in the same Storage daemon, the "status storage" command
1215 prompts for each different device, but they all give the same
1220 Item 39: Least recently used device selection for tape drives in autochanger.
1221 Date: 12 October 2009
1222 Origin: Thomas Carter <tcarter@memc.com>
1225 What: A better tape drive selection algorithm for multi-drive
1226 autochangers. The AUTOCHANGER class contains an array list of tape
1227 devices. When a tape drive is needed, this list is always searched in
1228 order. This causes lower number drives (specifically drive 0) to do a
1229 majority of the work with higher numbered drives possibly never being
1230 used. When a drive in an autochanger is reserved for use, its entry should
1231 be moved to the end of the list; this would give a rough LRU drive
1234 Why: The current implementation places a majority of use and wear on drive
1235 0 of a multi-drive autochanger.
1239 Item 40: Implement a Storage device like Amazon's S3.
1240 Date: 25 August 2008
1241 Origin: Soren Hansen <soren@ubuntu.com>
1242 Status: Not started.
1243 What: Enable the storage daemon to store backup data on Amazon's
1246 Why: Amazon's S3 is a cheap way to store data off-site.
1248 Notes: If we configure the Pool to put only one job per volume (they don't
1249 support append operation), and the volume size isn't to big (100MB?),
1250 it should be easy to adapt the disk-changer script to add get/put
1251 procedure with curl. So, the data would be safetly copied during the
1254 Cloud should be only used with Copy jobs, users should always have
1255 a copy of their data on their site.
1257 We should also think to have our own cache, trying always to have
1258 cloud volume on the local disk. (I don't know if users want to store
1259 100GB on cloud, so it shouldn't be a disk size problem). For example,
1260 if bacula want to recycle a volume, it will start by downloading the
1261 file to truncate it few seconds later, if we can avoid that...
1263 Item 41: Convert tray monitor on Windows to a stand alone program
1268 What: Separate Win32 tray monitor to be a separate program.
1270 Why: Vista does not allow SYSTEM services to interact with the
1271 desktop, so the current tray monitor does not work on Vista
1274 Notes: Requires communicating with the FD via the network (simulate
1275 a console connection).
1277 Item 42: Improve Bacula's tape and drive usage and cleaning management
1278 Date: 8 November 2005, November 11, 2005
1279 Origin: Adam Thornton <athornton at sinenomine dot net>,
1280 Arno Lehmann <al at its-lehmann dot de>
1284 1. Measure tape and drive usage (mostly implemented)
1285 2. Retiring a volume when too old or too many errors
1286 3. Handle cleaning and tape alerts.
1291 Item 43: Relabel disk volume after recycling
1292 Origin: Pasi Kärkkäinen <pasik@iki.fi>
1294 Status: Not implemented yet, no code written.
1296 What: The ability to relabel the disk volume (and thus rename the file on the
1297 disk) after it has been recycled. Useful when you have a single job
1298 per disk volume, and you use a custom Label format, for example:
1300 "${Client}-${Level}-${NumVols:p/4/0/r}-${Year}_${Month}_${Day}-${Hour}_${Minute}"
1302 Why: Disk volumes in Bacula get the label/filename when they are used for the
1303 first time. If you use recycling and custom label format like above,
1304 the disk volume name doesn't match the contents after it has been
1305 recycled. This feature makes it possible to keep the label/filename
1306 in sync with the content and thus makes it easy to check/monitor the
1307 backups from the shell and/or normal file management tools, because
1308 the filenames of the disk volumes match the content.
1310 Notes: The configuration option could be "Relabel after Recycling = Yes".
1314 ========= New items after last vote ====================
1317 Note to renumber items use:
1318 scripts/renumber_projects.pl projects >1
1321 ========= Add new items above this line =================
1324 ============= Empty Feature Request form ===========
1325 Item n: One line summary ...
1326 Date: Date submitted
1327 Origin: Name and email of originator.
1330 What: More detailed explanation ...
1332 Why: Why it is important ...
1334 Notes: Additional notes or features (omit if not used)
1335 ============== End Feature Request form ==============
1338 ========== Items put on hold by Kern ============================
1341 ========== Items completed in version 5.0.0 ====================
1342 *Item : 'restore' menu: enter a JobId, automatically select dependents
1343 *Item : Deletion of disk Volumes when pruned (partial -- truncate when pruned)
1344 *Item : Implement Base jobs
1345 *Item : Restore from volumes on multiple storage daemons
1346 *Item : Enable/disable compression depending on storage device (disk/tape)
1347 *Item : Cause daemons to use a specific IP address to source communications
1348 *Item : "Maximum Concurrent Jobs" for drives when used with changer device
1349 *Item : List InChanger flag when doing restore.
1350 *Item : Port bat to Win32
1351 *Item : An option to operate on all pools with update vol parameters