X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fprojects;h=fd6f1e7fcc48dd5f6182b542e70c771c29770aaa;hb=144d595c87ef7aab17d5e4d2190baa578c8ca49a;hp=ac00b840aada5e0b4dcda8489485388154b1f308;hpb=029e2474095160c466a269a2bd9203185b33c3d7;p=bacula%2Fbacula diff --git a/bacula/projects b/bacula/projects index ac00b840aa..fd6f1e7fcc 100644 --- a/bacula/projects +++ b/bacula/projects @@ -1,320 +1,386 @@ Projects: Bacula Projects Roadmap - Status updated 19 March 2008 - -Items Completed: + Status updated 8 August 2010 Summary: -Item 1: Accurate restoration of renamed/deleted files -Item 2: Allow FD to initiate a backup -Item 3: Merge multiple backups (Synthetic Backup or Consolidation) -Item 4: Implement Catalog directive for Pool resource in Director -Item 5: Add an item to the restore option where you can select a Pool -Item 6: Deletion of disk Volumes when pruned -Item 7: Implement Base jobs -Item 8: Implement Copy pools -Item 9: Scheduling syntax that permits more flexibility and options -Item 10: Message mailing based on backup types -Item 11: Cause daemons to use a specific IP address to source communications -Item 12: Add Plug-ins to the FileSet Include statements. -Item 13: Restore only file attributes (permissions, ACL, owner, group...) -Item 14: Add an override in Schedule for Pools based on backup types -Item 15: Implement more Python events and functions -Item 16: Allow inclusion/exclusion of files in a fileset by creation/mod times -Item 17: Automatic promotion of backup levels based on backup size -Item 18: Better control over Job execution -Item 19: Automatic disabling of devices -Item 20: An option to operate on all pools with update vol parameters -Item 21: Include timestamp of job launch in "stat clients" output -Item 22: Implement Storage daemon compression -Item 23: Improve Bacula's tape and drive usage and cleaning management -Item 24: Multiple threads in file daemon for the same job -Item 25: Archival (removal) of User Files to Tape - - -Item 1: Accurate restoration of renamed/deleted files - Date: 28 November 2005 - Origin: Martin Simmons (martin at lispworks dot com) - Status: +* => item complete + +Item 1: Ability to restart failed jobs +Item 2: SD redesign +Item 3: NDMP backup/restore +Item 4: SAP backup/restore +Item 5: Oracle backup/restore +Item 6: Zimbra and Zarafa backup/restore +Item 7: Include timestamp of job launch in "stat clients" output +Item 8: Include all conf files in specified directory +Item 9: Reduction of communications bandwidth for a backup +Item 10: Concurrent spooling and despooling within a single job. +Item 11: Start spooling even when waiting on tape +Item 12: Add ability to Verify any specified Job. +Item 13: Data encryption on storage daemon +Item 14: Possibilty to schedule Jobs on last Friday of the month +Item 15: Scheduling syntax that permits more flexibility and options +Item 16: Ability to defer Batch Insert to a later time +Item 17: Add MaxVolumeSize/MaxVolumeBytes to Storage resource +Item 18: Message mailing based on backup types +Item 19: Handle Windows Encrypted Files using Win raw encryption +Item 20: Job migration between different SDs +Item 19. Allow FD to initiate a backup +Item 21: Implement Storage daemon compression +Item 22: Ability to import/export Bacula database entities +Item 23: Implementation of running Job speed limit. +Item 24: Add an override in Schedule for Pools based on backup types +Item 25: Automatic promotion of backup levels based on backup size +Item 26: Allow FileSet inclusion/exclusion by creation/mod times +Item 27: Archival (removal) of User Files to Tape +Item 28: Ability to reconnect a disconnected comm line +Item 29: Multiple threads in file daemon for the same job +Item 30: Automatic disabling of devices +Item 31: Enable persistent naming/number of SQL queries +Item 32: Bacula Dir, FD and SD to support proxies +Item 33: Add Minumum Spool Size directive +Item 34: Command that releases all drives in an autochanger +Item 35: Run bscan on a remote storage daemon from within bconsole. +Item 36: Implement a Migration job type that will create a reverse +Item 37: Extend the verify code to make it possible to verify +Item 38: Separate "Storage" and "Device" in the bacula-dir.conf +Item 39: Least recently used device selection for tape drives in autochanger. +Item 40: Implement a Storage device like Amazon's S3. +Item 41: Convert tray monitor on Windows to a stand alone program +Item 42: Improve Bacula's tape and drive usage and cleaning management +Item 43: Relabel disk volume after recycling + + +Item 1: Ability to restart failed jobs + Date: 26 April 2009 + Origin: Kern/Eric + Status: + + What: Often jobs fail because of a communications line drop or max run time, + cancel, or some other non-critical problem. Currrently any data + saved is lost. This implementation should modify the Storage daemon + so that it saves all the files that it knows are completely backed + up to the Volume + + The jobs should then be marked as incomplete and a subsequent + Incremental Accurate backup will then take into account all the + previously saved job. + + Why: Avoids backuping data already saved. + + Notes: Requires Accurate to restart correctly. Must completed have a minimum + volume of data or files stored on Volume before enabling. + +Item 2: SD redesign + Date: 8 August 2010 + Origin: Kern + Status: + + What: Various ideas for redesigns planned for the SD: + 1. One thread per drive + 2. Design a class structure for all objects in the SD. + 3. Make Device into C++ classes for each device type + 4. Make Device have a proxy (front end intercept class) that will permit control over locking and changing the real device pointer. It can also permit delaying opening, so that we can adapt to having another program that tells us the Archive device name. + 5. Allow plugins to create new on the fly devices + 6. Separate SD volume manager + 7. Volume manager tells Bacula what drive or device to use for a given volume + + Why: It will simplify the SD, make it more modular, reduce locking + conflicts, and allow multiple buffer backups. + + +Item 3: NDMP backup/restore + Date: 8 August 2010 + Origin: Bacula Systems + Status: Enterprise only if implemented by Bacula Systems + + What: Backup/restore via NDMP -- most important NetApp compatibility + + + +Item 4: SAP backup/restore + Date: 8 August 2010 + Origin: Bacula Systems + Status: Enterprise only if implemented by Bacula Systems - What: When restoring a fileset for a specified date (including "most - recent"), Bacula should give you exactly the files and directories - that existed at the time of the last backup prior to that date. + What: Backup/restore SAP databases (MaxDB, Oracle, possibly DB2) - Currently this only works if the last backup was a Full backup. - When the last backup was Incremental/Differential, files and - directories that have been renamed or deleted since the last Full - backup are not currently restored correctly. Ditto for files with - extra/fewer hard links than at the time of the last Full backup. - Why: Incremental/Differential would be much more useful if this worked. - Notes: Merging of multiple backups into a single one seems to - rely on this working, otherwise the merged backups will not be - truly equivalent to a Full backup. +Item 5: Oracle backup/restore + Date: 8 August 2010 + Origin: Bacula Systems + Status: Enterprise only if implemented by Bacula Systems - Note: Kern: notes shortened. This can be done without the need for - inodes. It is essentially the same as the current Verify job, - but one additional database record must be written, which does - not need any database change. + What: Backup/restore Oracle databases - Notes: Kern: see if we can correct restoration of directories if - replace=ifnewer is set. Currently, if the directory does not - exist, a "dummy" directory is created, then when all the files - are updated, the dummy directory is newer so the real values - are not updated. -Item 2: Allow FD to initiate a backup - Origin: Frank Volf (frank at deze dot org) - Date: 17 November 2005 +Item 6: Zimbra and Zarafa backup/restore + Date: 8 August 2010 + Origin: Bacula Systems + Status: Enterprise only if implemented by Bacula Systems + + What: Backup/restore for Zimbra and Zarafa + + + +Item 7: Include timestamp of job launch in "stat clients" output + Origin: Mark Bergman + Date: Tue Aug 22 17:13:39 EDT 2006 Status: - What: Provide some means, possibly by a restricted console that - allows a FD to initiate a backup, and that uses the connection - established by the FD to the Director for the backup so that - a Director that is firewalled can do the backup. + What: The "stat clients" command doesn't include any detail on when + the active backup jobs were launched. - Why: Makes backup of laptops much easier. + Why: Including the timestamp would make it much easier to decide whether + a job is running properly. + Notes: It may be helpful to have the output from "stat clients" formatted + more like that from "stat dir" (and other commands), in a column + format. The per-client information that's currently shown (level, + client name, JobId, Volume, pool, device, Files, etc.) is good, but + somewhat hard to parse (both programmatically and visually), + particularly when there are many active clients. -Item 3: Merge multiple backups (Synthetic Backup or Consolidation) - Origin: Marc Cousin and Eric Bollengier - Date: 15 November 2005 - Status: - What: A merged backup is a backup made without connecting to the Client. - It would be a Merge of existing backups into a single backup. - In effect, it is like a restore but to the backup medium. - - For instance, say that last Sunday we made a full backup. Then - all week long, we created incremental backups, in order to do - them fast. Now comes Sunday again, and we need another full. - The merged backup makes it possible to do instead an incremental - backup (during the night for instance), and then create a merged - backup during the day, by using the full and incrementals from - the week. The merged backup will be exactly like a full made - Sunday night on the tape, but the production interruption on the - Client will be minimal, as the Client will only have to send - incrementals. - - In fact, if it's done correctly, you could merge all the - Incrementals into single Incremental, or all the Incrementals - and the last Differential into a new Differential, or the Full, - last differential and all the Incrementals into a new Full - backup. And there is no need to involve the Client. - - Why: The benefit is that : - - the Client just does an incremental ; - - the merged backup on tape is just as a single full backup, - and can be restored very fast. - - This is also a way of reducing the backup data since the old - data can then be pruned (or not) from the catalog, possibly - allowing older volumes to be recycled - -Item 4: Implement Catalog directive for Pool resource in Director - Origin: Alan Davis adavis@ruckus.com - Date: 6 March 2007 - Status: Submitted - - What: The current behavior is for the director to create all pools - found in the configuration file in all catalogs. Add a - Catalog directive to the Pool resource to specify which - catalog to use for each pool definition. +Item 8: Include all conf files in specified directory +Date: 18 October 2008 +Origin: Database, Lda. Maputo, Mozambique +Contact:Cameron Smith / cameron.ord@database.co.mz +Status: New request + +What: A directive something like "IncludeConf = /etc/bacula/subconfs" Every + time Bacula Director restarts or reloads, it will walk the given + directory (non-recursively) and include the contents of any files + therein, as though they were appended to bacula-dir.conf + +Why: Permits simplified and safer configuration for larger installations with + many client PCs. Currently, through judicious use of JobDefs and + similar directives, it is possible to reduce the client-specific part of + a configuration to a minimum. The client-specific directives can be + prepared according to a standard template and dropped into a known + directory. However it is still necessary to add a line to the "master" + (bacula-dir.conf) referencing each new file. This exposes the master to + unnecessary risk of accidental mistakes and makes automation of adding + new client-confs, more difficult (it is easier to automate dropping a + file into a dir, than rewriting an existing file). Ken has previously + made a convincing argument for NOT including Bacula's core configuration + in an RDBMS, but I believe that the present request is a reasonable + extension to the current "flat-file-based" configuration philosophy. - Why: This allows different catalogs to have different pool - attributes and eliminates the side-effect of adding - pools to catalogs that don't need/use them. - - Notes: Kern: I think this is relatively easy to do, and it is really - a pre-requisite to a number of the Copy pool, ... projects - that are listed here. - -Item 5: Add an item to the restore option where you can select a Pool - Origin: kshatriyak at gmail dot com - Date: 1/1/2006 +Notes: There is NO need for any special syntax to these files. They should + contain standard directives which are simply "inlined" to the parent + file as already happens when you explicitly reference an external file. + +Notes: (kes) this can already be done with scripting + From: John Jorgensen + The bacula-dir.conf at our site contains these lines: + + # + # Include subfiles associated with configuration of clients. + # They define the bulk of the Clients, Jobs, and FileSets. + # + @|"sh -c 'for f in /etc/bacula/clientdefs/*.conf ; do echo @${f} ; done'" + + and when we get a new client, we just put its configuration into + a new file called something like: + + /etc/bacula/clientdefs/clientname.conf + + + + +Item 9: Reduction of communications bandwidth for a backup + Date: 14 October 2008 + Origin: Robin O'Leary (Equiinet) + Status: + + What: Using rdiff techniques, Bacula could significantly reduce + the network data transfer volume to do a backup. + + Why: Faster backup across the Internet + + Notes: This requires retaining certain data on the client during a Full + backup that will speed up subsequent backups. + + +Item 10: Concurrent spooling and despooling within a single job. +Date: 17 nov 2009 +Origin: Jesper Krogh +Status: NEW +What: When a job has spooling enabled and the spool area size is + less than the total volumes size the storage daemon will: + 1) Spool to spool-area + 2) Despool to tape + 3) Go to 1 if more data to be backed up. + + Typical disks will serve data with a speed of 100MB/s when + dealing with large files, network it typical capable of doing 115MB/s + (GbitE). Tape drives will despool with 50-90MB/s (LTO3) 70-120MB/s + (LTO4) depending on compression and data. + + As bacula currently works it'll hold back data from the client until + de-spooling is done, now matter if the spool area can handle another + block of data. Say given a FileSet of 4TB and a spool-area of 100GB and + a Maximum Job Spool Size set to 50GB then above sequence could be + changed to allow to spool to the other 50GB while despooling the first + 50GB and not holding back the client while doing it. As above numbers + show, depending on tape-drive and disk-arrays this potentially leads to + a cut of the backup-time of 50% for the individual jobs. + + Real-world example, backing up 112.6GB (large files) to LTO4 tapes + (despools with ~75MB/s, data is gzipped on the remote filesystem. + Maximum Job Spool Size = 8GB + + Current: + Size: 112.6GB + Elapsed time (total time): 46m 15s => 2775s + Despooling time: 25m 41s => 1541s (55%) + Spooling time: 20m 34s => 1234s (45%) + Reported speed: 40.58MB/s + Spooling speed: 112.6GB/1234s => 91.25MB/s + Despooling speed: 112.6GB/1541s => 73.07MB/s + + So disk + net can "keep up" with the LTO4 drive (in this test) + + Prosed change would effectively make the backup run in the "despooling + time" 1541s giving a reduction to 55% of the total run time. + + In the situation where the individual job cannot keep up with LTO-drive + spooling enables efficient multiplexing of multiple concurrent jobs onto + the same drive. + +Why: When dealing with larger volumes the general utillization of the + network/disk is important to maximize in order to be able to run a full + backup over a weekend. Current work-around is to split the FileSet in + smaller FileSet and Jobs but that leads to more configuration mangement + and is harder to review for completeness. Subsequently it makes restores + more complex. + + + +Item 11: Start spooling even when waiting on tape + Origin: Tobias Barth + Date: 25 April 2008 Status: - What: In the restore option (Select the most recent backup for a - client) it would be useful to add an option where you can limit - the selection to a certain pool. + What: If a job can be spooled to disk before writing it to tape, it should + be spooled immediately. Currently, bacula waits until the correct + tape is inserted into the drive. + + Why: It could save hours. When bacula waits on the operator who must insert + the correct tape (e.g. a new tape or a tape from another media + pool), bacula could already prepare the spooled data in the spooling + directory and immediately start despooling when the tape was + inserted by the operator. + + 2nd step: Use 2 or more spooling directories. When one directory is + currently despooling, the next (on different disk drives) could + already be spooling the next data. + + Notes: I am using bacula 2.2.8, which has none of those features + implemented. + + +Item 12: Add ability to Verify any specified Job. +Date: 17 January 2008 +Origin: portrix.net Hamburg, Germany. +Contact: Christian Sabelmann +Status: 70% of the required Code is part of the Verify function since v. 2.x + + What: + The ability to tell Bacula which Job should verify instead of + automatically verify just the last one. + + Why: + It is sad that such a powerfull feature like Verify Jobs + (VolumeToCatalog) is restricted to be used only with the last backup Job + of a client. Actual users who have to do daily Backups are forced to + also do daily Verify Jobs in order to take advantage of this useful + feature. This Daily Verify after Backup conduct is not always desired + and Verify Jobs have to be sometimes scheduled. (Not necessarily + scheduled in Bacula). With this feature Admins can verify Jobs once a + Week or less per month, selecting the Jobs they want to verify. This + feature is also not to difficult to implement taking in account older bug + reports about this feature and the selection of the Job to be verified. + + Notes: For the verify Job, the user could select the Job to be verified + from a List of the latest Jobs of a client. It would also be possible to + verify a certain volume. All of these would naturaly apply only for + Jobs whose file information are still in the catalog. + - Why: When using cloned jobs, most of the time you have 2 pools - a - disk pool and a tape pool. People who have 2 pools would like to - select the most recent backup from disk, not from tape (tape - would be only needed in emergency). However, the most recent - backup (which may just differ a second from the disk backup) may - be on tape and would be selected. The problem becomes bigger if - you have a full and differential - the most "recent" full backup - may be on disk, while the most recent differential may be on tape - (though the differential on disk may differ even only a second or - so). Bacula will complain that the backups reside on different - media then. For now the only solution now when restoring things - when you have 2 pools is to manually search for the right - job-id's and enter them by hand, which is a bit fault tolerant. +Item 13: Data encryption on storage daemon + Origin: Tobias Barth + Date: 04 February 2009 + Status: new - Notes: Kern: This is a nice idea. It could also be the way to support - Jobs that have been Copied (similar to migration, but not yet - implemented). + What: The storage demon should be able to do the data encryption that can + currently be done by the file daemon. + Why: This would have 2 advantages: + 1) one could encrypt the data of unencrypted tapes by doing a + migration job + 2) the storage daemon would be the only machine that would have + to keep the encryption keys. + Notes from Landon: + As an addendum to the feature request, here are some crypto + implementation details I wrote up regarding SD-encryption back in Jan + 2008: + http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg28860.html -Item 6: Deletion of disk Volumes when pruned - Date: Nov 25, 2005 - Origin: Ross Boylan (edited - by Kern) - Status: - What: Provide a way for Bacula to automatically remove Volumes - from the filesystem, or optionally to truncate them. - Obviously, the Volume must be pruned prior removal. - Why: This would allow users more control over their Volumes and - prevent disk based volumes from consuming too much space. +Item 14: Possibilty to schedule Jobs on last Friday of the month +Origin: Carsten Menke +Date: 02 March 2008 +Status: - Notes: The following two directives might do the trick: + What: Currently if you want to run your monthly Backups on the last + Friday of each month this is only possible with workarounds (e.g + scripting) (As some months got 4 Fridays and some got 5 Fridays) + The same is true if you plan to run your yearly Backups on the + last Friday of the year. It would be nice to have the ability to + use the builtin scheduler for this. - Volume Data Retention =