+
+Item 8: Implement Copy pools
+ Date: 27 November 2005
+ Origin: David Boyes (dboyes at sinenomine dot net)
+ Status:
+
+ What: I would like Bacula to have the capability to write copies
+ of backed-up data on multiple physical volumes selected
+ from different pools without transferring the data
+ multiple times, and to accept any of the copy volumes
+ as valid for restore.
+
+ Why: In many cases, businesses are required to keep offsite
+ copies of backup volumes, or just wish for simple
+ protection against a human operator dropping a storage
+ volume and damaging it. The ability to generate multiple
+ volumes in the course of a single backup job allows
+ customers to simple check out one copy and send it
+ offsite, marking it as out of changer or otherwise
+ unavailable. Currently, the library and magazine
+ management capability in Bacula does not make this process
+ simple.
+
+ Restores would use the copy of the data on the first
+ available volume, in order of Copy pool chain definition.
+
+ This is also a major scalability issue -- as the number of
+ clients increases beyond several thousand, and the volume
+ of data increases, transferring the data multiple times to
+ produce additional copies of the backups will become
+ physically impossible due to transfer speed
+ issues. Generating multiple copies at server side will
+ become the only practical option.
+
+ How: I suspect that this will require adding a multiplexing
+ SD that appears to be a SD to a specific FD, but 1-n FDs
+ to the specific back end SDs managing the primary and copy
+ pools. Storage pools will also need to acquire parameters
+ to define the pools to be used for copies.
+
+ Notes: I would commit some of my developers' time if we can agree
+ on the design and behavior.
+
+ Notes: Additional notes from David:
+ I think there's two areas where new configuration would be needed.
+
+ 1) Identify a "SD mux" SD (specify it in the config just like a normal
+ SD. The SD configuration would need something like a "Daemon Type =
+ Normal/Mux" keyword to identify it as a multiplexor. (The director code
+ would need modification to add the ability to do the multiple session
+ setup, but the impact of the change would be new code that was invoked
+ only when a SDmux is needed).
+
+ 2) Additional keywords in the Pool definition to identify the need to
+ create copies. Each pool would acquire a Copypool= attribute (may be
+ repeated to generate more than one copy. 3 is about the practical limit,
+ but no point in hardcoding that).
+
+ Example:
+ Pool {
+ Name = Primary
+ Pool Type = Backup
+ Copypool = Copy1
+ Copypool = OffsiteCopy2
+ }
+
+ where Copy1 and OffsiteCopy2 are valid pools.
+
+ In terms of function (shorthand):
+ Backup job X is defined normally, specifying pool Primary as the pool to
+ use. Job gets scheduled, and Bacula starts scheduling resources.
+ Scheduler looks at pool definition for Primary, sees that there are a
+ non-zero number of copypool keywords. The director then connects to an
+ available SDmux, passes it the pool ids for Primary, Copy1, and
+ OffsiteCopy2 and waits. SDmux then goes out and reserves devices and
+ volumes in the normal SDs that serve Primary, Copy1 and OffsiteCopy2.
+ When all are ready, the SDmux signals ready back to the director, and
+ the FD is given the address of the SDmux as the SD to communicate with.
+ Backup proceeds normally, with the SDmux duplicating blocks to each
+ connected normal SD, and returning ready when all defined copies have
+ been written. At EOJ, FD shuts down connection with SDmux, which closes
+ down the normal SD connections and goes back to an idle state.
+ SDmux does not update database; normal SDs do (noting that file is
+ present on each volume it has been written to).
+
+ On restore, director looks for the volume containing the file in pool
+ Primary first, then Copy1, then OffsiteCopy2. If the volume holding the
+ file in pool Primary is missing or busy (being written in another job,
+ etc), or one of the volumes from the copypool list that have the file in
+ question is already mounted and ready for some reason, use it to do the
+ restore, else mount one of the copypool volumes and proceed.
+
+
+Item 9: Scheduling syntax that permits more flexibility and options
+ Date: 15 December 2006
+ Origin: Gregory Brauer (greg at wildbrain dot com) and
+ Florian Schnabel <florian.schnabel at docufy dot de>
+ Status:
+
+ What: Currently, Bacula only understands how to deal with weeks of the
+ month or weeks of the year in schedules. This makes it impossible
+ to do a true weekly rotation of tapes. There will always be a
+ discontinuity that will require disruptive manual intervention at
+ least monthly or yearly because week boundaries never align with
+ month or year boundaries.
+
+ A solution would be to add a new syntax that defines (at least)
+ a start timestamp, and repetition period.
+
+ An easy option to skip a certain job on a certain date.
+
+
+ Why: Rotated backups done at weekly intervals are useful, and Bacula
+ cannot currently do them without extensive hacking.
+
+ You could then easily skip tape backups on holidays. Especially
+ if you got no autochanger and can only fit one backup on a tape
+ that would be really handy, other jobs could proceed normally
+ and you won't get errors that way.
+
+
+ Notes: Here is an example syntax showing a 3-week rotation where full
+ Backups would be performed every week on Saturday, and an
+ incremental would be performed every week on Tuesday. Each
+ set of tapes could be removed from the loader for the following
+ two cycles before coming back and being reused on the third
+ week. Since the execution times are determined by intervals
+ from a given point in time, there will never be any issues with
+ having to adjust to any sort of arbitrary time boundary. In
+ the example provided, I even define the starting schedule
+ as crossing both a year and a month boundary, but the run times
+ would be based on the "Repeat" value and would therefore happen
+ weekly as desired.
+
+
+ Schedule {
+ Name = "Week 1 Rotation"
+ #Saturday. Would run Dec 30, Jan 20, Feb 10, etc.
+ Run {
+ Options {
+ Type = Full
+ Start = 2006-12-30 01:00
+ Repeat = 3w
+ }
+ }
+ #Tuesday. Would run Jan 2, Jan 23, Feb 13, etc.
+ Run {
+ Options {
+ Type = Incremental
+ Start = 2007-01-02 01:00
+ Repeat = 3w
+ }
+ }
+ }
+
+ Schedule {
+ Name = "Week 2 Rotation"
+ #Saturday. Would run Jan 6, Jan 27, Feb 17, etc.
+ Run {
+ Options {
+ Type = Full
+ Start = 2007-01-06 01:00
+ Repeat = 3w
+ }
+ }
+ #Tuesday. Would run Jan 9, Jan 30, Feb 20, etc.
+ Run {
+ Options {
+ Type = Incremental
+ Start = 2007-01-09 01:00
+ Repeat = 3w
+ }
+ }
+ }
+
+ Schedule {
+ Name = "Week 3 Rotation"
+ #Saturday. Would run Jan 13, Feb 3, Feb 24, etc.
+ Run {
+ Options {
+ Type = Full
+ Start = 2007-01-13 01:00
+ Repeat = 3w
+ }
+ }
+ #Tuesday. Would run Jan 16, Feb 6, Feb 27, etc.
+ Run {
+ Options {
+ Type = Incremental
+ Start = 2007-01-16 01:00
+ Repeat = 3w
+ }
+ }
+ }
+
+ Notes: Kern: I have merged the previously separate project of skipping
+ jobs (via Schedule syntax) into this.
+
+
+Item 10: Message mailing based on backup types
+ Origin: Evan Kaufman <evan.kaufman@gmail.com>
+ Date: January 6, 2006
+ Status:
+
+ What: In the "Messages" resource definitions, allowing messages
+ to be mailed based on the type (backup, restore, etc.) and level
+ (full, differential, etc) of job that created the originating
+ message(s).
+
+ Why: It would, for example, allow someone's boss to be emailed
+ automatically only when a Full Backup job runs, so he can
+ retrieve the tapes for offsite storage, even if the IT dept.
+ doesn't (or can't) explicitly notify him. At the same time, his
+ mailbox wouldnt be filled by notifications of Verifies, Restores,
+ or Incremental/Differential Backups (which would likely be kept
+ onsite).
+
+ Notes: One way this could be done is through additional message types, for example:
+
+ Messages {
+ # email the boss only on full system backups
+ Mail = boss@mycompany.com = full, !incremental, !differential, !restore,
+ !verify, !admin
+ # email us only when something breaks
+ MailOnError = itdept@mycompany.com = all
+ }
+
+ Notes: Kern: This should be rather trivial to implement.
+
+
+Item 11: Cause daemons to use a specific IP address to source communications
+ Origin: Bill Moran <wmoran@collaborativefusion.com>
+ Date: 18 Dec 2006
+ Status:
+ What: Cause Bacula daemons (dir, fd, sd) to always use the ip address
+ specified in the [DIR|DF|SD]Addr directive as the source IP
+ for initiating communication.
+ Why: On complex networks, as well as extremely secure networks, it's
+ not unusual to have multiple possible routes through the network.
+ Often, each of these routes is secured by different policies
+ (effectively, firewalls allow or deny different traffic depending
+ on the source address)
+ Unfortunately, it can sometimes be difficult or impossible to
+ represent this in a system routing table, as the result is
+ excessive subnetting that quickly exhausts available IP space.
+ The best available workaround is to provide multiple IPs to
+ a single machine that are all on the same subnet. In order
+ for this to work properly, applications must support the ability
+ to bind outgoing connections to a specified address, otherwise
+ the operating system will always choose the first IP that
+ matches the required route.
+ Notes: Many other programs support this. For example, the following
+ can be configured in BIND:
+ query-source address 10.0.0.1;
+ transfer-source 10.0.0.2;
+ Which means queries from this server will always come from
+ 10.0.0.1 and zone transfers will always originate from
+ 10.0.0.2.
+
+
+Item 12: Add Plug-ins to the FileSet Include statements.