3 Bacula Projects Roadmap
4 Status updated 26 April 2009
7 Item 1: Allow FD to initiate a backup
8 Item 2: Ability to restart failed jobs
9 Item 3: Port bat to Win32
10 Item 4: Convert Bacula existing tray monitor on Windows to a stand alone program
11 Item 5: Ability to import/export Bacula database entities
12 Item 6: Ability to defer Batch Insert to a later time
13 Item 7: List InChanger flag when doing restore.
14 Item 8: Deletion of disk Volumes when pruned
15 Item 9: Implement Base jobs
16 Item 10: Scheduling syntax that permits more flexibility and options
17 Item 11: Reduction of communications bandwidth for a backup
18 Item 12: Bacula Dir, FD and SD to support proxies
19 Item 13: Message mailing based on backup types
20 Item 14: Ability to reconnect a disconnected comm line
21 Item 15: Include timestamp of job launch in "stat clients" output
22 Item 16: Add an override in Schedule for Pools based on backup types
23 Item 17: Automatic promotion of backup levels based on backup size
24 Item 18: Allow inclusion/exclusion of files in a fileset by creation/mod times
25 Item 19: Archival (removal) of User Files to Tape
26 Item 20: Include all conf files in specified directory
27 Item 21: Implement an interface between Bacula and Amazon's S3.
28 Item 22: Enable/disable compression depending on storage device (disk/tape)
29 Item 23: Backup and Restore of Windows Encrypted Files using Win raw encryption
30 Item 24: Data encryption on storage daemon
31 Item 25: "Maximum Concurrent Jobs" for drives when used with changer device
32 Item 26: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
33 Item 27: Start spooling even when waiting on tape
34 Item 28: Enable persistent naming/number of SQL queries
35 Item 29: Implementation of running Job speed limit.
36 Item 30: Restore from volumes on multiple storage daemons
37 Item 31: 'restore' menu: enter a JobId, automatically select dependents
38 Item 32: Possibilty to schedule Jobs on last Friday of the month
39 Item 33: Add Minumum Spool Size directive
40 Item 34: Cause daemons to use a specific IP address to source communications
41 Item 35: Add ability to Verify any specified Job.
42 Item 36: Automatic disabling of devices
43 Item 37: An option to operate on all pools with update vol parameters
44 Item 38: Implement Storage daemon compression
45 Item 39: Improve Bacula's tape and drive usage and cleaning management
46 Item 40: Multiple threads in file daemon for the same job
49 Item 1: Allow FD to initiate a backup
50 Origin: Frank Volf (frank at deze dot org)
51 Date: 17 November 2005
54 What: Provide some means, possibly by a restricted console that
55 allows a FD to initiate a backup, and that uses the connection
56 established by the FD to the Director for the backup so that
57 a Director that is firewalled can do the backup.
58 Why: Makes backup of laptops much easier.
60 Item 2: Ability to restart failed jobs
65 What: Often jobs fail because of a communications line drop or max run time,
66 cancel, or some other non-critical problem. Currrently any data
67 saved is lost. This implementation should modify the Storage daemon
68 so that it saves all the files that it knows are completely backed
71 The jobs should then be marked as incomplete and a subsequent
72 Incremental Accurate backup will then take into account all the
75 Why: Avoids backuping data already saved.
77 Notes: Requires Accurate to restart correctly. Must completed have a minimum
78 volume of data or files stored on Volume before enabling.
80 Item 3: Port bat to Win32
85 What: Make bat run on Win32/64.
87 Why: To have GUI on Windows
91 Item 4: Convert Bacula existing tray monitor on Windows to a stand alone program
96 What: Separate Win32 tray monitor to be a separate program.
98 Why: Vista does not allow SYSTEM services to interact with the
99 desktop, so the current tray monitor does not work on Vista
102 Notes: Requires communicating with the FD via the network (simulate
103 a console connection).
107 Item 5: Ability to import/export Bacula database entities
112 What: Create a Bacula ASCII SQL database independent format that permits
113 importing and exporting database catalog Job entities.
115 Why: For achival, database clustering, tranfer to other databases
118 Notes: Job selection should be by Job, time, Volume, Client, Pool and possibly
122 Item 6: Ability to defer Batch Insert to a later time
127 What: Instead of doing a Job Batch Insert at the end of the Job
128 which might create resource contention with lots of Job,
129 defer the insert to a later time.
131 Why: Permits to focus on getting the data on the Volume and
132 putting the metadata into the Catalog outside the backup
135 Notes: Will use the proposed Bacula ASCII database import/export
136 format (i.e. dependent on the import/export entities project).
139 Item 7: List InChanger flag when doing restore.
140 Origin: Jesper Krogh<jesper@krogh.cc>
144 What: When doing a restore the restore selection dialog ends by telling stuff
146 The job will require the following
147 Volume(s) Storage(s) SD Device(s)
148 ===========================================================================
159 When having an autochanger, it would be really nice with an inChanger
160 column so the operator knew if this restore job would stop waiting for
161 operator intervention. This is done just by selecting the inChanger flag
162 from the catalog and printing it in a seperate column.
165 Why: This would help getting large restores through minimizing the
166 time spent waiting for operator to drop by and change tapes in the library.
168 Notes: [Kern] I think it would also be good to have the Slot as well,
169 or some indication that Bacula thinks the volume is in the autochanger
170 because it depends on both the InChanger flag and the Slot being
174 Item 8: Deletion of disk Volumes when pruned
176 Origin: Ross Boylan <RossBoylan at stanfordalumni dot org> (edited
180 What: Provide a way for Bacula to automatically remove Volumes
181 from the filesystem, or optionally to truncate them.
182 Obviously, the Volume must be pruned prior removal.
184 Why: This would allow users more control over their Volumes and
185 prevent disk based volumes from consuming too much space.
187 Notes: The following two directives might do the trick:
189 Volume Data Retention = <time period>
190 Remove Volume After = <time period>
192 The migration project should also remove a Volume that is
193 migrated. This might also work for tape Volumes.
195 Notes: (Kern). The data fields to control this have been added
196 to the new 3.0.0 database table structure.
200 Item 9: Implement Base jobs
201 Date: 28 October 2005
205 What: A base job is sort of like a Full save except that you
206 will want the FileSet to contain only files that are
207 unlikely to change in the future (i.e. a snapshot of
208 most of your system after installing it). After the
209 base job has been run, when you are doing a Full save,
210 you specify one or more Base jobs to be used. All
211 files that have been backed up in the Base job/jobs but
212 not modified will then be excluded from the backup.
213 During a restore, the Base jobs will be automatically
214 pulled in where necessary.
216 Why: This is something none of the competition does, as far as
217 we know (except perhaps BackupPC, which is a Perl program that
218 saves to disk only). It is big win for the user, it
219 makes Bacula stand out as offering a unique
220 optimization that immediately saves time and money.
221 Basically, imagine that you have 100 nearly identical
222 Windows or Linux machine containing the OS and user
223 files. Now for the OS part, a Base job will be backed
224 up once, and rather than making 100 copies of the OS,
225 there will be only one. If one or more of the systems
226 have some files updated, no problem, they will be
227 automatically restored.
229 Notes: Huge savings in tape usage even for a single machine.
230 Will require more resources because the DIR must send
231 FD a list of files/attribs, and the FD must search the
232 list and compare it for each file to be saved.
235 Item 10: Scheduling syntax that permits more flexibility and options
236 Date: 15 December 2006
237 Origin: Gregory Brauer (greg at wildbrain dot com) and
238 Florian Schnabel <florian.schnabel at docufy dot de>
241 What: Currently, Bacula only understands how to deal with weeks of the
242 month or weeks of the year in schedules. This makes it impossible
243 to do a true weekly rotation of tapes. There will always be a
244 discontinuity that will require disruptive manual intervention at
245 least monthly or yearly because week boundaries never align with
246 month or year boundaries.
248 A solution would be to add a new syntax that defines (at least)
249 a start timestamp, and repetition period.
251 An easy option to skip a certain job on a certain date.
254 Why: Rotated backups done at weekly intervals are useful, and Bacula
255 cannot currently do them without extensive hacking.
257 You could then easily skip tape backups on holidays. Especially
258 if you got no autochanger and can only fit one backup on a tape
259 that would be really handy, other jobs could proceed normally
260 and you won't get errors that way.
263 Notes: Here is an example syntax showing a 3-week rotation where full
264 Backups would be performed every week on Saturday, and an
265 incremental would be performed every week on Tuesday. Each
266 set of tapes could be removed from the loader for the following
267 two cycles before coming back and being reused on the third
268 week. Since the execution times are determined by intervals
269 from a given point in time, there will never be any issues with
270 having to adjust to any sort of arbitrary time boundary. In
271 the example provided, I even define the starting schedule
272 as crossing both a year and a month boundary, but the run times
273 would be based on the "Repeat" value and would therefore happen
278 Name = "Week 1 Rotation"
279 #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
283 Start = 2006-12-30 01:00
287 #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
291 Start = 2007-01-02 01:00
298 Name = "Week 2 Rotation"
299 #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
303 Start = 2007-01-06 01:00
307 #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
311 Start = 2007-01-09 01:00
318 Name = "Week 3 Rotation"
319 #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
323 Start = 2007-01-13 01:00
327 #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
331 Start = 2007-01-16 01:00
337 Notes: Kern: I have merged the previously separate project of skipping
338 jobs (via Schedule syntax) into this.
340 Item 11: Reduction of communications bandwidth for a backup
341 Date: 14 October 2008
342 Origin: Robin O'Leary (Equiinet)
345 What: Using rdiff techniques, Bacula could significantly reduce
346 the network data transfer volume to do a backup.
348 Why: Faster backup across the Internet
350 Notes: This requires retaining certain data on the client during a Full
351 backup that will speed up subsequent backups.
354 Item 12: Bacula Dir, FD and SD to support proxies
355 Origin: Karl Grindley @ MIT Lincoln Laboratory <kgrindley at ll dot mit dot edu>
359 What: Support alternate methods for nailing up a TCP session such
360 as SOCKS5, SOCKS4 and HTTP (CONNECT) proxies. Such a feature
361 would allow tunneling of bacula traffic in and out of proxied
364 Why: Currently, bacula is architected to only function on a flat network, with
365 no barriers or limitations. Due to the large configuration states of
366 any network and the infinite configuration where file daemons and
367 storage daemons may sit in relation to one another, bacula often is
368 not usable on a network where filtered or air-gaped networks exist.
369 While often solutions such as ACL modifications to firewalls or port
370 redirection via SNAT or DNAT will solve the issue, often however,
371 these solutions are not adequate or not allowed by hard policy.
373 In an air-gapped network with only a highly locked down proxy services
374 are provided (SOCKS4/5 and/or HTTP and/or SSH outbound) ACLs or
375 iptable rules will not work.
377 Notes: Director resource tunneling: This configuration option to utilize a
378 proxy to connect to a client should be specified in the client
379 resource Client resource tunneling: should be configured in the client
380 resource in the director config file? Or configured on the bacula-fd
381 configuration file on the fd host itself? If the ladder, this would
382 allow only certain clients to use a proxy, where others do not when
383 establishing the TCP connection to the storage server.
385 Also worth noting, there are other 3rd party, light weight apps that
386 could be utilized to bootstrap this. Instead of sockifing bacula
387 itself, use an external program to broker proxy authentication, and
388 connection to the remote host. OpenSSH does this by using the
389 "ProxyCommand" syntax in the client configuration and uses stdin and
390 stdout to the command. Connect.c is a very popular one.
391 (http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html).
392 One could also possibly use stunnel, netcat, etc.
395 Item 13: Message mailing based on backup types
396 Origin: Evan Kaufman <evan.kaufman@gmail.com>
397 Date: January 6, 2006
400 What: In the "Messages" resource definitions, allowing messages
401 to be mailed based on the type (backup, restore, etc.) and level
402 (full, differential, etc) of job that created the originating
405 Why: It would, for example, allow someone's boss to be emailed
406 automatically only when a Full Backup job runs, so he can
407 retrieve the tapes for offsite storage, even if the IT dept.
408 doesn't (or can't) explicitly notify him. At the same time, his
409 mailbox wouldnt be filled by notifications of Verifies, Restores,
410 or Incremental/Differential Backups (which would likely be kept
413 Notes: One way this could be done is through additional message types, for example:
416 # email the boss only on full system backups
417 Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
419 # email us only when something breaks
420 MailOnError = itdept@mycompany.com = all
423 Notes: Kern: This should be rather trivial to implement.
425 Item 14: Ability to reconnect a disconnected comm line
430 What: Often jobs fail because of a communications line drop. In that
431 case, Bacula should be able to reconnect to the other daemon and
434 Why: Avoids backuping data already saved.
436 Notes: *Very* complicated from a design point of view because of authenication.
439 Item 15: Include timestamp of job launch in "stat clients" output
440 Origin: Mark Bergman <mark.bergman@uphs.upenn.edu>
441 Date: Tue Aug 22 17:13:39 EDT 2006
444 What: The "stat clients" command doesn't include any detail on when
445 the active backup jobs were launched.
447 Why: Including the timestamp would make it much easier to decide whether
448 a job is running properly.
450 Notes: It may be helpful to have the output from "stat clients" formatted
451 more like that from "stat dir" (and other commands), in a column
452 format. The per-client information that's currently shown (level,
453 client name, JobId, Volume, pool, device, Files, etc.) is good, but
454 somewhat hard to parse (both programmatically and visually),
455 particularly when there are many active clients.
460 Item 16: Add an override in Schedule for Pools based on backup types
462 Origin: Chad Slater <chad.slater@clickfox.com>
465 What: Adding a FullStorage=BigTapeLibrary in the Schedule resource
466 would help those of us who use different storage devices for different
467 backup levels cope with the "auto-upgrade" of a backup.
469 Why: Assume I add several new devices to be backed up, i.e. several
470 hosts with 1TB RAID. To avoid tape switching hassles, incrementals are
471 stored in a disk set on a 2TB RAID. If you add these devices in the
472 middle of the month, the incrementals are upgraded to "full" backups,
473 but they try to use the same storage device as requested in the
474 incremental job, filling up the RAID holding the differentials. If we
475 could override the Storage parameter for full and/or differential
476 backups, then the Full job would use the proper Storage device, which
477 has more capacity (i.e. a 8TB tape library.
479 Item 17: Automatic promotion of backup levels based on backup size
480 Date: 19 January 2006
481 Origin: Adam Thornton <athornton@sinenomine.net>
484 What: Other backup programs have a feature whereby it estimates the space
485 that a differential, incremental, and full backup would take. If
486 the difference in space required between the scheduled level and the
487 next level up is beneath some user-defined critical threshold, the
488 backup level is bumped to the next type. Doing this minimizes the
489 number of volumes necessary during a restore, with a fairly minimal
490 cost in backup media space.
492 Why: I know at least one (quite sophisticated and smart) user for whom the
493 absence of this feature is a deal-breaker in terms of using Bacula;
494 if we had it it would eliminate the one cool thing other backup
495 programs can do and we can't (at least, the one cool thing I know
498 Item 18: Allow inclusion/exclusion of files in a fileset by creation/mod times
499 Origin: Evan Kaufman <evan.kaufman@gmail.com>
500 Date: January 11, 2006
503 What: In the vein of the Wild and Regex directives in a Fileset's
504 Options, it would be helpful to allow a user to include or exclude
505 files and directories by creation or modification times.
507 You could factor the Exclude=yes|no option in much the same way it
508 affects the Wild and Regex directives. For example, you could exclude
509 all files modified before a certain date:
513 Modified Before = ####
516 Or you could exclude all files created/modified since a certain date:
520 Created Modified Since = ####
523 The format of the time/date could be done several ways, say the number
524 of seconds since the epoch:
525 1137008553 = Jan 11 2006, 1:42:33PM # result of `date +%s`
527 Or a human readable date in a cryptic form:
528 20060111134233 = Jan 11 2006, 1:42:33PM # YYYYMMDDhhmmss
530 Why: I imagine a feature like this could have many uses. It would
531 allow a user to do a full backup while excluding the base operating
532 system files, so if I installed a Linux snapshot from a CD yesterday,
533 I'll *exclude* all files modified *before* today. If I need to
534 recover the system, I use the CD I already have, plus the tape backup.
535 Or if, say, a Windows client is hit by a particularly corrosive
536 virus, and I need to *exclude* any files created/modified *since* the
539 Notes: Of course, this feature would work in concert with other
540 in/exclude rules, and wouldnt override them (or each other).
542 Notes: The directives I'd imagine would be along the lines of
543 "[Created] [Modified] [Before|Since] = <date>".
544 So one could compare against 'ctime' and/or 'mtime', but ONLY 'before'
549 Item 19: Archival (removal) of User Files to Tape
551 Origin: Ray Pengelly [ray at biomed dot queensu dot ca
554 What: The ability to archive data to storage based on certain parameters
555 such as age, size, or location. Once the data has been written to
556 storage and logged it is then pruned from the originating
557 filesystem. Note! We are talking about user's files and not
560 Why: This would allow fully automatic storage management which becomes
561 useful for large datastores. It would also allow for auto-staging
562 from one media type to another.
564 Example 1) Medical imaging needs to store large amounts of data.
565 They decide to keep data on their servers for 6 months and then put
566 it away for long term storage. The server then finds all files
567 older than 6 months writes them to tape. The files are then removed
570 Example 2) All data that hasn't been accessed in 2 months could be
571 moved from high-cost, fibre-channel disk storage to a low-cost
572 large-capacity SATA disk storage pool which doesn't have as quick of
573 access time. Then after another 6 months (or possibly as one
574 storage pool gets full) data is migrated to Tape.
577 Item 20: Include all conf files in specified directory
578 Date: 18 October 2008
579 Origin: Database, Lda. Maputo, Mozambique
580 Contact:Cameron Smith / cameron.ord@database.co.mz
583 What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every
584 time Bacula Director restarts or reloads, it will walk the given
585 directory (non-recursively) and include the contents of any files
586 therein, as though they were appended to bacula-dir.conf
588 Why: Permits simplified and safer configuration for larger installations with
589 many client PCs. Currently, through judicious use of JobDefs and
590 similar directives, it is possible to reduce the client-specific part of
591 a configuration to a minimum. The client-specific directives can be
592 prepared according to a standard template and dropped into a known
593 directory. However it is still necessary to add a line to the "master"
594 (bacula-dir.conf) referencing each new file. This exposes the master to
595 unnecessary risk of accidental mistakes and makes automation of adding
596 new client-confs, more difficult (it is easier to automate dropping a
597 file into a dir, than rewriting an existing file). Ken has previously
598 made a convincing argument for NOT including Bacula's core configuration
599 in an RDBMS, but I believe that the present request is a reasonable
600 extension to the current "flat-file-based" configuration philosophy.
602 Notes: There is NO need for any special syntax to these files. They should
603 contain standard directives which are simply "inlined" to the parent
604 file as already happens when you explicitly reference an external file.
606 Notes: (kes) this can already be done with scripting
607 From: John Jorgensen <jorgnsn@lcd.uregina.ca>
608 The bacula-dir.conf at our site contains these lines:
611 # Include subfiles associated with configuration of clients.
612 # They define the bulk of the Clients, Jobs, and FileSets.
614 @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'"
616 and when we get a new client, we just put its configuration into
617 a new file called something like:
619 /etc/bacula/clientdefs/clientname.conf
624 Item 21: Implement an interface between Bacula and Amazon's S3.
626 Origin: Soren Hansen <soren@ubuntu.com>
628 What: Enable the storage daemon to store backup data on Amazon's
631 Why: Amazon's S3 is a cheap way to store data off-site. Current
632 ways to integrate Bacula and S3 involve storing all the data
633 locally and syncing them to S3, and manually fetching them
634 again when they're needed. This is very cumbersome.
637 Item 22: Enable/disable compression depending on storage device (disk/tape)
638 Origin: Ralf Gross ralf-lists@ralfgross.de
640 Status: Initial Request
642 What: Add a new option to the storage resource of the director. Depending
643 on this option, compression will be enabled/disabled for a device.
645 Why: If different devices (disks/tapes) are used for full/diff/incr
646 backups, software compression will be enabled for all backups
647 because of the FileSet compression option. For backup to tapes
648 wich are able to do hardware compression this is not desired.
652 http://news.gmane.org/gmane.comp.sysutils.backup.bacula.devel/cutoff=11124
653 It must be clear to the user, that the FileSet compression option
654 must still be enabled use compression for a backup job at all.
655 Thus a name for the new option in the director must be
658 Notes: KES I think the Storage definition should probably override what
659 is in the Job definition or vice-versa, but in any case, it must
663 Item 31: Backup and Restore of Windows Encrypted Files using Win raw encryption
664 Origin: Michael Mohr, SAG Mohr.External@infineon.com
665 Date: 22 February 2008
666 Origin: Alex Ehrlich (Alex.Ehrlich-at-mail.ee)
670 What: Make it possible to backup and restore Encypted Files from and to
671 Windows systems without the need to decrypt it by using the raw
672 encryption functions API (see:
673 http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
674 that is provided for that reason by Microsoft.
675 If a file ist encrypted could be examined by evaluating the
676 FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes
678 For each file backed up or restored by FD on Windows, check if
679 the file is encrypted; if so then use OpenEncryptedFileRaw,
680 ReadEncryptedFileRaw, WriteEncryptedFileRaw,
681 CloseEncryptedFileRaw instead of BackupRead and BackupWrite
684 Why: Without the usage of this interface the fd-daemon running
685 under the system account can't read encypted Files because
686 the key needed for the decrytion is missed by them. As a result
687 actually encrypted files are not backed up
688 by bacula and also no error is shown while missing these files.
690 Notes: Using xxxEncryptedFileRaw API would allow to backup and
691 restore EFS-encrypted files without decrypting their data.
692 Note that such files cannot be restored "portably" (at least,
693 easily) but they would be restoreable to a different (or
694 reinstalled) Win32 machine; the restore would require setup
695 of a EFS recovery agent in advance, of course, and this shall
696 be clearly reflected in the documentation, but this is the
697 normal Windows SysAdmin's business.
698 When "portable" backup is requested the EFS-encrypted files
699 shall be clearly reported as errors.
700 See MSDN on the "Backup and Restore of Encrypted Files" topic:
701 http://msdn.microsoft.com/en-us/library/aa363783.aspx
702 Maybe the EFS support requires a new flag in the database for
704 Unfortunately, the implementation is not as straightforward as
705 1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
706 requiring some FD code rewrite to work with
707 encrypted-file-related callback functions.
709 Item 24: Data encryption on storage daemon
710 Origin: Tobias Barth <tobias.barth at web-arts.com>
711 Date: 04 February 2009
714 What: The storage demon should be able to do the data encryption that can currently be done by the file daemon.
716 Why: This would have 2 advantages: 1) one could encrypt the data of unencrypted tapes by doing a migration job, and 2) the storage daemon would be the only machine that would have to keep the encryption keys.
719 As an addendum to the feature request, here are some crypto
720 implementation details I wrote up regarding SD-encryption back in Jan
722 http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html
726 Item 25: "Maximum Concurrent Jobs" for drives when used with changer device
727 Origin: Ralf Gross ralf-lists <at> ralfgross.de
729 Status: Initial Request
731 What: respect the "Maximum Concurrent Jobs" directive in the _drives_
732 Storage section in addition to the changer section
734 Why: I have a 3 drive changer where I want to be able to let 3 concurrent
735 jobs run in parallel. But only one job per drive at the same time.
736 Right now I don't see how I could limit the number of concurrent jobs
737 per drive in this situation.
739 Notes: Using different priorities for these jobs lead to problems that other
740 jobs are blocked. On the user list I got the advice to use the "Prefer Mounted
741 Volumes" directive, but Kern advised against using "Prefer Mounted
742 Volumes" in an other thread:
743 http://article.gmane.org/gmane.comp.sysutils.backup.bacula.devel/11876/
745 In addition I'm not sure if this would be the same as respecting the
746 drive's "Maximum Concurrent Jobs" setting.
758 Maximum Concurrent Jobs = 3
762 Name = Neo4100-LTO4-D1
766 Device = ULTRIUM-TD4-D1
768 Maximum Concurrent Jobs = 1
773 The "Maximum Concurrent Jobs = 1" directive in the drive's section is ignored.
775 Item 26: Add MaxVolumeSize/MaxVolumeBytes statement to Storage resource
776 Origin: Bastian Friedrich <bastian.friedrich@collax.com>
780 What: SD has a "Maximum Volume Size" statement, which is deprecated and
781 superseded by the Pool resource statement "Maximum Volume Bytes".
782 It would be good if either statement could be used in Storage
785 Why: Pools do not have to be restricted to a single storage type/device;
786 thus, it may be impossible to define Maximum Volume Bytes in the
787 Pool resource. The old MaxVolSize statement is deprecated, as it
788 is SD side only. I am using the same pool for different devices.
790 Notes: State of idea currently unknown. Storage resources in the dir
791 config currently translate to very slim catalog entries; these
792 entries would require extensions to implement what is described
793 here. Quite possibly, numerous other statements that are currently
794 available in Pool resources could be used in Storage resources too
797 Item 27: Start spooling even when waiting on tape
798 Origin: Tobias Barth <tobias.barth@web-arts.com>
802 What: If a job can be spooled to disk before writing it to tape, it should
803 be spooled immediately. Currently, bacula waits until the correct
804 tape is inserted into the drive.
806 Why: It could save hours. When bacula waits on the operator who must insert
807 the correct tape (e.g. a new tape or a tape from another media
808 pool), bacula could already prepare the spooled data in the spooling
809 directory and immediately start despooling when the tape was
810 inserted by the operator.
812 2nd step: Use 2 or more spooling directories. When one directory is
813 currently despooling, the next (on different disk drives) could
814 already be spooling the next data.
816 Notes: I am using bacula 2.2.8, which has none of those features
819 Item 28: Enable persistent naming/number of SQL queries
825 Change the parsing of the query.sql file and the query command so that
826 queries are named/numbered by a fixed value, not their order in the
831 One of the real strengths of bacula is the ability to query the
832 database, and the fact that complex queries can be saved and
833 referenced from a file is very powerful. However, the choice
834 of query (both for interactive use, and by scripting input
835 to the bconsole command) is completely dependent on the order
836 within the query.sql file. The descriptve labels are helpful for
837 interactive use, but users become used to calling a particular
838 query "by number", or may use scripts to execute queries. This
839 presents a problem if the number or order of queries in the file
842 If the query.sql file used the numeric tags as a real value (rather
843 than a comment), then users could have a higher confidence that they
844 are executing the intended query, that their local changes wouldn't
845 conflict with future bacula upgrades.
847 For scripting, it's very important that the intended query is
848 what's actually executed. The current method of parsing the
849 query.sql file discourages scripting because the addition or
850 deletion of queries within the file will require corresponding
851 changes to scripts. It may not be obvious to users that deleting
852 query "17" in the query.sql file will require changing all
853 references to higher numbered queries. Similarly, when new
854 bacula distributions change the number of "official" queries,
855 user-developed queries cannot simply be appended to the file
856 without also changing any references to those queries in scripts
857 or procedural documentation, etc.
859 In addition, using fixed numbers for queries would encourage more
860 user-initiated development of queries, by supporting conventions
863 queries numbered 1-50 are supported/developed/distributed by
864 with official bacula releases
866 queries numbered 100-200 are community contributed, and are
867 related to media management
869 queries numbered 201-300 are community contributed, and are
870 related to checksums, finding duplicated files across
871 different backups, etc.
873 queries numbered 301-400 are community contributed, and are
874 related to backup statistics (average file size, size per
875 client per backup level, time for all clients by backup level,
876 storage capacity by media type, etc.)
878 queries numbered 500-999 are locally created
881 Alternatively, queries could be called by keyword (tag), rather
884 Item 29: Implementation of running Job speed limit.
885 Origin: Alex F, alexxzell at yahoo dot com
886 Date: 29 January 2009
888 What: I noticed the need for an integrated bandwidth limiter for
889 running jobs. It would be very useful just to specify another
890 field in bacula-dir.conf, like speed = how much speed you wish
891 for that specific job to run at
893 Why: Because of a couple of reasons. First, it's very hard to implement a
894 traffic shaping utility and also make it reliable. Second, it is very
895 uncomfortable to have to implement these apps to, let's say 50 clients
896 (including desktops, servers). This would also be unreliable because you
897 have to make sure that the apps are properly working when needed; users
898 could also disable them (accidentally or not). It would be very useful
899 to provide Bacula this ability. All information would be centralized,
900 you would not have to go to 50 different clients in 10 different
901 locations for configuration; eliminating 3rd party additions help in
902 establishing efficiency. Would also avoid bandwidth congestion,
903 especially where there is little available.
905 Item 30: Restore from volumes on multiple storage daemons
906 Origin: Graham Keeling (graham@equiinet.com)
910 What: The ability to restore from volumes held by multiple storage daemons
911 would be very useful.
913 Why: It is useful to be able to backup to any number of different storage
914 daemons. For example, your first storage daemon may run out of space, so you
915 switch to your second and carry on. Bacula will currently let you do this.
916 However, once you come to restore, bacula cannot cope when volumes on different
917 storage daemons are required.
919 Notes: The director knows that more than one storage daemon is needed, as
920 bconsole outputs something like the following table.
922 The job will require the following
923 Volume(s) Storage(s) SD Device(s)
924 ===========================================================================
926 backup-0001 Disk 1 Disk 1.0
927 backup-0002 Disk 2 Disk 2.0
929 However, the bootstrap file that it creates gets sent to the first storage
930 daemon only, which then stalls for a long time, 'waiting for a mount request'
931 for the volume that it doesn't have.
932 The bootstrap file contains no knowledge of the storage daemon.
933 Under the current design:
935 The director connects to the storage daemon, and gets an sd_auth_key.
936 The director then connects to the file daemon, and gives it the
937 sd_auth_key with the 'jobcmd'.
938 (restoring of files happens)
939 The director does a 'wait_for_storage_daemon_termination()'.
940 The director waits for the file daemon to indicate the end of the job.
944 The director connects to the file daemon.
945 Then, for each storage daemon in the .bsr file... {
946 The director connects to the storage daemon, and gets an sd_auth_key.
947 The director then connects to the file daemon, and gives it the
948 sd_auth_key with the 'storaddr' command.
949 (restoring of files happens)
950 The director does a 'wait_for_storage_daemon_termination()'.
951 The director waits for the file daemon to indicate the end of the
952 work on this storage.
954 The director tells the file daemon that there are no more storages to contact.
955 The director waits for the file daemon to indicate the end of the job.
957 As you can see, each restore between the file daemon and storage daemon is
958 handled in the same way that it is currently handled, using the same method
959 for authentication, except that the sd_auth_key is moved from the 'jobcmd' to
960 the 'storaddr' command - where it logically belongs.
963 Item 31: 'restore' menu: enter a JobId, automatically select dependents
964 Origin: Graham Keeling (graham@equiinet.com)
969 What: Add to the bconsole 'restore' menu the ability to select a job
970 by JobId, and have bacula automatically select all the dependent jobs.
972 Why: Currently, you either have to...
973 a) laboriously type in a date that is greater than the date of the backup that
974 you want and is less than the subsequent backup (bacula then figures out the
976 b) manually figure out all the JobIds that you want and laboriously type them
978 It would be extremely useful (in a programmatical sense, as well as for humans)
979 to be able to just give it a single JobId and let bacula do the hard work (work
980 that it already knows how to do).
982 Notes (Kern): I think this should either be modified to have Bacula print
983 a list of dates that the user can choose from as is done in bwx-console and
984 bat or the name of this command must be carefully chosen so that the user
985 clearly understands that the JobId is being used to specify what Job and the
986 date to which he wishes the restore to happen.
990 Item 32: Possibilty to schedule Jobs on last Friday of the month
991 Origin: Carsten Menke <bootsy52 at gmx dot net>
995 What: Currently if you want to run your monthly Backups on the last
996 Friday of each month this is only possible with workarounds (e.g
997 scripting) (As some months got 4 Fridays and some got 5 Fridays)
998 The same is true if you plan to run your yearly Backups on the
999 last Friday of the year. It would be nice to have the ability to
1000 use the builtin scheduler for this.
1002 Why: In many companies the last working day of the week is Friday (or
1003 Saturday), so to get the most data of the month onto the monthly
1004 tape, the employees are advised to insert the tape for the
1005 monthly backups on the last friday of the month.
1007 Notes: To give this a complete functionality it would be nice if the
1008 "first" and "last" Keywords could be implemented in the
1009 scheduler, so it is also possible to run monthy backups at the
1010 first friday of the month and many things more. So if the syntax
1011 would expand to this {first|last} {Month|Week|Day|Mo-Fri} of the
1012 {Year|Month|Week} you would be able to run really flexible jobs.
1014 To got a certain Job run on the last Friday of the Month for example one could
1017 Run = pool=Monthly last Fri of the Month at 23:50
1021 Run = pool=Yearly last Fri of the Year at 23:50
1023 ## Certain Jobs the last Week of a Month
1025 Run = pool=LastWeek last Week of the Month at 23:50
1027 ## Monthly Backup on the last day of the month
1029 Run = pool=Monthly last Day of the Month at 23:50
1031 Item 33: Add Minumum Spool Size directive
1033 Origin: Frank Sweetser <fs@wpi.edu>
1035 What: Add a new SD directive, "minimum spool size" (or similar). This
1036 directive would specify a minimum level of free space available for
1037 spooling. If the unused spool space is less than this level, any
1038 new spooling requests would be blocked as if the "maximum spool
1039 size" threshold had bee reached. Already spooling jobs would be
1040 unaffected by this directive.
1042 Why: I've been bitten by this scenario a couple of times:
1044 Assume a maximum spool size of 100M. Two concurrent jobs, A and B,
1045 are both running. Due to timing quirks and previously running jobs,
1046 job A has used 99.9M of space in the spool directory. While A is
1047 busy despooling to disk, B is happily using the remaining 0.1M of
1048 spool space. This ends up in a spool/despool sequence every 0.1M of
1049 data. In addition to fragmenting the data on the volume far more
1050 than was necessary, in larger data sets (ie, tens or hundreds of
1051 gigabytes) it can easily produce multi-megabyte report emails!
1054 Item 34: Cause daemons to use a specific IP address to source communications
1055 Origin: Bill Moran <wmoran@collaborativefusion.com>
1058 What: Cause Bacula daemons (dir, fd, sd) to always use the ip address
1059 specified in the [DIR|DF|SD]Addr directive as the source IP
1060 for initiating communication.
1061 Why: On complex networks, as well as extremely secure networks, it's
1062 not unusual to have multiple possible routes through the network.
1063 Often, each of these routes is secured by different policies
1064 (effectively, firewalls allow or deny different traffic depending
1065 on the source address)
1066 Unfortunately, it can sometimes be difficult or impossible to
1067 represent this in a system routing table, as the result is
1068 excessive subnetting that quickly exhausts available IP space.
1069 The best available workaround is to provide multiple IPs to
1070 a single machine that are all on the same subnet. In order
1071 for this to work properly, applications must support the ability
1072 to bind outgoing connections to a specified address, otherwise
1073 the operating system will always choose the first IP that
1074 matches the required route.
1075 Notes: Many other programs support this. For example, the following
1076 can be configured in BIND:
1077 query-source address 10.0.0.1;
1078 transfer-source 10.0.0.2;
1079 Which means queries from this server will always come from
1080 10.0.0.1 and zone transfers will always originate from
1085 Item 35: Add ability to Verify any specified Job.
1086 Date: 17 January 2008
1087 Origin: portrix.net Hamburg, Germany.
1088 Contact: Christian Sabelmann
1089 Status: 70% of the required Code is part of the Verify function since v. 2.x
1092 The ability to tell Bacula which Job should verify instead of
1093 automatically verify just the last one.
1096 It is sad that such a powerfull feature like Verify Jobs
1097 (VolumeToCatalog) is restricted to be used only with the last backup Job
1098 of a client. Actual users who have to do daily Backups are forced to
1099 also do daily Verify Jobs in order to take advantage of this useful
1100 feature. This Daily Verify after Backup conduct is not always desired
1101 and Verify Jobs have to be sometimes scheduled. (Not necessarily
1102 scheduled in Bacula). With this feature Admins can verify Jobs once a
1103 Week or less per month, selecting the Jobs they want to verify. This
1104 feature is also not to difficult to implement taking in account older bug
1105 reports about this feature and the selection of the Job to be verified.
1107 Notes: For the verify Job, the user could select the Job to be verified
1108 from a List of the latest Jobs of a client. It would also be possible to
1109 verify a certain volume. All of these would naturaly apply only for
1110 Jobs whose file information are still in the catalog.
1112 Item 36: Automatic disabling of devices
1114 Origin: Peter Eriksson <peter at ifm.liu dot se>
1117 What: After a configurable amount of fatal errors with a tape drive
1118 Bacula should automatically disable further use of a certain
1119 tape drive. There should also be "disable"/"enable" commands in
1120 the "bconsole" tool.
1122 Why: On a multi-drive jukebox there is a possibility of tape drives
1123 going bad during large backups (needing a cleaning tape run,
1124 tapes getting stuck). It would be advantageous if Bacula would
1125 automatically disable further use of a problematic tape drive
1126 after a configurable amount of errors has occurred.
1128 An example: I have a multi-drive jukebox (6 drives, 380+ slots)
1129 where tapes occasionally get stuck inside the drive. Bacula will
1130 notice that the "mtx-changer" command will fail and then fail
1131 any backup jobs trying to use that drive. However, it will still
1132 keep on trying to run new jobs using that drive and fail -
1133 forever, and thus failing lots and lots of jobs... Since we have
1134 many drives Bacula could have just automatically disabled
1135 further use of that drive and used one of the other ones
1138 Item 37: An option to operate on all pools with update vol parameters
1139 Origin: Dmitriy Pinchukov <absh@bossdev.kiev.ua>
1140 Date: 16 August 2006
1141 Status: Patch made by Nigel Stepp
1143 What: When I do update -> Volume parameters -> All Volumes
1144 from Pool, then I have to select pools one by one. I'd like
1145 console to have an option like "0: All Pools" in the list of
1148 Why: I have many pools and therefore unhappy with manually
1149 updating each of them using update -> Volume parameters -> All
1150 Volumes from Pool -> pool #.
1152 Item 38: Implement Storage daemon compression
1153 Date: 18 December 2006
1154 Origin: Vadim A. Umanski , e-mail umanski@ext.ru
1156 What: The ability to compress backup data on the SD receiving data
1157 instead of doing that on client sending data.
1158 Why: The need is practical. I've got some machines that can send
1159 data to the network 4 or 5 times faster than compressing
1160 them (I've measured that). They're using fast enough SCSI/FC
1161 disk subsystems but rather slow CPUs (ex. UltraSPARC II).
1162 And the backup server has got a quite fast CPUs (ex. Dual P4
1163 Xeons) and quite a low load. When you have 20, 50 or 100 GB
1164 of raw data - running a job 4 to 5 times faster - that
1165 really matters. On the other hand, the data can be
1166 compressed 50% or better - so losing twice more space for
1167 disk backup is not good at all. And the network is all mine
1168 (I have a dedicated management/provisioning network) and I
1169 can get as high bandwidth as I need - 100Mbps, 1000Mbps...
1170 That's why the server-side compression feature is needed!
1173 Item 39: Improve Bacula's tape and drive usage and cleaning management
1174 Date: 8 November 2005, November 11, 2005
1175 Origin: Adam Thornton <athornton at sinenomine dot net>,
1176 Arno Lehmann <al at its-lehmann dot de>
1179 What: Make Bacula manage tape life cycle information, tape reuse
1180 times and drive cleaning cycles.
1182 Why: All three parts of this project are important when operating
1184 We need to know which tapes need replacement, and we need to
1185 make sure the drives are cleaned when necessary. While many
1186 tape libraries and even autoloaders can handle all this
1187 automatically, support by Bacula can be helpful for smaller
1188 (older) libraries and single drives. Limiting the number of
1189 times a tape is used might prevent tape errors when using
1190 tapes until the drives can't read it any more. Also, checking
1191 drive status during operation can prevent some failures (as I
1192 [Arno] had to learn the hard way...)
1194 Notes: First, Bacula could (and even does, to some limited extent)
1195 record tape and drive usage. For tapes, the number of mounts,
1196 the amount of data, and the time the tape has actually been
1197 running could be recorded. Data fields for Read and Write
1198 time and Number of mounts already exist in the catalog (I'm
1199 not sure if VolBytes is the sum of all bytes ever written to
1200 that volume by Bacula). This information can be important
1201 when determining which media to replace. The ability to mark
1202 Volumes as "used up" after a given number of write cycles
1203 should also be implemented so that a tape is never actually
1204 worn out. For the tape drives known to Bacula, similar
1205 information is interesting to determine the device status and
1206 expected life time: Time it's been Reading and Writing, number
1207 of tape Loads / Unloads / Errors. This information is not yet
1208 recorded as far as I [Arno] know. A new volume status would
1209 be necessary for the new state, like "Used up" or "Worn out".
1210 Volumes with this state could be used for restores, but not
1211 for writing. These volumes should be migrated first (assuming
1212 migration is implemented) and, once they are no longer needed,
1213 could be moved to a Trash pool.
1215 The next step would be to implement a drive cleaning setup.
1216 Bacula already has knowledge about cleaning tapes. Once it
1217 has some information about cleaning cycles (measured in drive
1218 run time, number of tapes used, or calender days, for example)
1219 it can automatically execute tape cleaning (with an
1220 autochanger, obviously) or ask for operator assistance loading
1223 The final step would be to implement TAPEALERT checks not only
1224 when changing tapes and only sending the information to the
1225 administrator, but rather checking after each tape error,
1226 checking on a regular basis (for example after each tape
1227 file), and also before unloading and after loading a new tape.
1228 Then, depending on the drives TAPEALERT state and the known
1229 drive cleaning state Bacula could automatically schedule later
1230 cleaning, clean immediately, or inform the operator.
1232 Implementing this would perhaps require another catalog change
1233 and perhaps major changes in SD code and the DIR-SD protocol,
1234 so I'd only consider this worth implementing if it would
1235 actually be used or even needed by many people.
1237 Implementation of these projects could happen in three distinct
1238 sub-projects: Measuring Tape and Drive usage, retiring
1239 volumes, and handling drive cleaning and TAPEALERTs.
1241 Item 40: Multiple threads in file daemon for the same job
1242 Date: 27 November 2005
1243 Origin: Ove Risberg (Ove.Risberg at octocode dot com)
1246 What: I want the file daemon to start multiple threads for a backup
1247 job so the fastest possible backup can be made.
1249 The file daemon could parse the FileSet information and start
1250 one thread for each File entry located on a separate
1253 A confiuration option in the job section should be used to
1254 enable or disable this feature. The confgutration option could
1255 specify the maximum number of threads in the file daemon.
1257 If the theads could spool the data to separate spool files
1258 the restore process will not be much slower.
1260 Why: Multiple concurrent backups of a large fileserver with many
1261 disks and controllers will be much faster.
1263 ========= End items voted on May 2009 ==================
1265 ========= New items after last vote ====================
1267 Item 1: Relabel disk volume after recycling
1268 Origin: Pasi Kärkkäinen <pasik@iki.fi>
1270 Status: Not implemented yet, no code written.
1272 What: The ability to relabel the disk volume (and thus rename the file on the disk)
1273 after it has been recycled. Useful when you have a single job per disk volume,
1274 and you use a custom Label format, for example:
1275 Label Format = "${Client}-${Level}-${NumVols:p/4/0/r}-${Year}_${Month}_${Day}-${Hour}_${Minute}"
1277 Why: Disk volumes in Bacula get the label/filename when they are used for the first time.
1278 If you use recycling and custom label format like above, the disk
1279 volume name doesn't match the contents after it has been recycled.
1280 This feature makes it possible to keep the label/filename in sync
1281 with the content and thus makes it easy to check/monitor the backups
1282 from the shell and/or normal file management tools, because the filenames
1283 of the disk volumes match the content.
1285 Notes: The configuration option could be "Relabel after Recycling = Yes".
1288 ========= Add new items above this line =================
1291 ============= Empty Feature Request form ===========
1292 Item n: One line summary ...
1293 Date: Date submitted
1294 Origin: Name and email of originator.
1297 What: More detailed explanation ...
1299 Why: Why it is important ...
1301 Notes: Additional notes or features (omit if not used)
1302 ============== End Feature Request form ==============
1305 ========== Items put on hold by Kern ============================