Kern's ToDo List
- 31 March 2004
+ 22 July 2004
+
+1.35 Items to do:
+- Add bscan to four-concurrent-jobs regression.
+- Do tape alerts -- see tapealert.txt
+- Document a get out of jail procedure if everything breaks if
+ you lost/broke the Catalog -- do the same for "I know my
+ file is there how do I get it back?".
+- When passwords do not match, print message that points the
+ user to the doc.
+- Make Verify jobs require exclusive use of Volume as Restore
+ jobs do.
+- Perhaps add read/write programs and/or plugins to FileSets.
+- Look at adding Client run command that will use the
+ port opened by the client.
+- Fix find_device in stored/dircmd.c:462
+- Add new DCR calling sequences everywhere in SD.
+ This will permit simultaneous use of multiple
+ devices by a single job.
+- bscan does not put first of two volumes back with all info in
+ bscan-test.
+- Fix restore ++++ that get intermingled with "Building directory tree"
+- Solve the termcap.h problem on Solaris configure.
+- Fix ./configure to handle installed SQLite
+- Test Win32 errno handling.
Documentation to do: (any release a little bit at a time)
- Document query file format.
changing tape settings while a job is running.)
- Lookup HP cleaning recommendations.
- Lookup HP tape replacement recommendations (see trouble shooting autochanger)
+- Add more obvious documentation of @ for conf files.
+- Document doing table repair
Testing to do: (painful)
- Test drive polling!
- Test if rewind at end of tape waits for tape to rewind.
- Test cancel at EOM.
-For 1.33 Testing/Documentation:
-- Figure out how to use ssh or stunnel to protect Bacula communications.
- Add Dan's work to manual See ssl.txt
+For 1.37 Testing/Documentation:
- Add db check test to regression. Test each function like delete,
purge, ...
- If you use restore replace=never, the directory attributes for
non-existent directories will not be restored properly.
-1.34 Cleanup
+Wish list:
+- Figure out how to make named console messages go only to that
+ console and to the non-restricted console (new console class?).
+- Make restricted console prompt for password if *ask* is set or
+ perhaps if password is undefined.
+- Implement "from ISO-date/time every x hours/days/weeks/months" in
+ schedules.
+- Compaction of Disk space by "migrating" Volumes that have pruned
+ Jobs (what criteria? size, #jobs, time).
+- Add prune all command
+- Document fact that purge can destroy a part of a restore by purging
+ one volume while others remain valid -- perhaps mark Jobs.
- Add multiple-media-types.txt
- look at mxt-changer.html
-
-
-For version 1.35:
+- Make ? do a help command (no return needed).
+- Implement restore directory.
+- Add All Local Partitions = yes to new style saves.
+- Document streams and how to implement them.
+- Possibly implement "Ensure Full Backup = yes" looks for a failed full backup
+ and upgrades the current backup if one exists.
+- Check that barcode reading and update slots scan works.
+- Try not to re-backup a file if a new hard link is added.
+- Add feature to backup hard links only, but not the data.
+- Add "All Local = yes" option to save to include all local partitions.
+- Fix stream handling to be simpler.
+- Add Priority and Bootstrap to Run a Job.
+- Eliminate Restore "Run Restore Job" prompt by allowing new "run command
+ to be issued"
+- Remove View FileSet button from Run a Job dialog.
+- Don't count higher level non-saved directories in Files expected
+ in restore report.
+- Handle prompt for restore job at end of Restore command.
+- Add display of total selected files to Restore window.
+- Add tree pane to left of window.
+- Add progress meter.
+- Polling does not work for restore. It tries a number of times,
+ gives up, and crashes the SD.
+- Lock jcr_chain when doing attach/detach in acquire.c
+- Add assert in free_jcr if attach/detach chain active.
+- Max wait time or max run time causes seg fault -- see runtime-bug.txt
+- Document writing to a CD/DVD with Bacula.
+- Add check for tape alerts.
+- Add a "base" package to the window installer for pthreadsVCE.dll
+ which is needed by all packages.
+- Add message to user to check for fixed block size when the forward
+ space test fails in btape.
+- When unmarking a directory check if all files below are unmarked and
+ then remove the + flag -- in the restore tree.
- Possibly implement: Action = Unmount Device="TapeDrive1" in Admin jobs.
- Setup lrrd graphs: (http://www.linpro.no/projects/lrrd/) Mike Acar.
- Revisit the question of multiple Volumes (disk) on a single device.
- Investigate adding Mac Resource Forks.
- Finish work on Gnome restore GUI.
- Fix "llist jobid=xx" where no fileset or client exists.
+- Check pruning of restore jobs.
- From Chris Hull:
it seems to be complaining about 12:00pm which should be a valid 12
hour time. I changed the time to 11:59am and everything works fine.
Also 12:00am works fine. 0:00pm also works (which I don't think
should). None of the values 12:00pm - 12:59pm work for that matter.
- Add level to estimate command.
+- For each job type (Admin, Restore, ...) require only the really necessary
+ fields.
- Fix option 2 of restore -- list where file is backed up -- require Client,
then list last 20 backups.
- Add all pools in Dir conf to DB also update them to catch changed
in Job, Client, and Pool, with precedence such that Job overrides Client
which in turn overrides Pool.
- Print a message when a job starts if the conf file is not current.
-- To pass Include 1 or two letter commands
- I Name Include name - first record
- B Name Base name - repeat
- R "xxx" Regexp
- W "xxx" Wild Card
- E zzz Exclude expression (wild card)
- P "plugin" Plugin
- D "reader" Reader program
- T "writer" Writer program
- O Options In current commpressed format
- (compression, signature, onefs, recurse, sparse,
- replace, verify options, ...)
- N End option set
- B BaseName Start second option set any letter
- ...
- E
- F Number Number of filenames to follow
- <file-name>
- B Name
- ...
- N End option set
- F Number Number of filenames to follow
- <file-name>
- ...
- Spooling ideas taken from Volker Sauer's and other's emails:
> IMHO job spooling should be turned on
> woorkstations to be shut down overnight to save power.
>
-- From Terry Manderson <terry@apnic.net>
- jobdefs { # new structure
- name = "monthlyUnixBoxen"
- type = backup
- level = full
- schedule = monthly
- storage = DLT
- messages = Standard
- pool = MonthlyPool
- priority = 10
- }
-
- job {
- name = "wakame"
- jobdefs = "genericUnixSet"
- client = wakame-fd
- }
-
- job {
- name = "durian"
- jobdefs = "genericUnixSet"
- client = durian-fd
- }
-
- job {
- name = "soy"
- jobdefs = "UnixDevelBoxSet"
- client = soy-fd
- }
-
-
- Autolabel should be specified by DIR instead of SD.
- Storage daemon
- Add media capacity
VolSessionId and VolSessionTime.
=========================================================
-
-
-=============================================================
-
- Request For Comments For File Backup Options
- 10 November 2002
-
-Subject: File Backup Options
-
-Problem:
- A few days ago, a Bacula user who is backing up to file volumes and
- using compression asked if it was possible to suppress compressing
- all .gz files since it was a waste of CPU time. Although Bacula
- currently permits using different options (compression, ...) on
- a directory by directory basis, it cannot do it on a file by
- file basis, which is clearly what was desired.
-
-Proposed Implementation:
- To solve this problem, I propose the following:
-
- - Add a new Director resource type called Options.
-
- - The Options resource will have records for all
- options that can currently be specified on the Include record
- (in a FileSet). Examples below.
-
- - The Options resource will permit an exclude option as well
- as a number of additional options.
-
- - The heart of the Options resource is the ability to
- supply any number of Match records which specify POSIX
- regular expressions. These Match regular expressions are
- applied to the fully qualified filename (path and all). If
- one matches, then the Options will be used.
-
- - When an Match specification matches an included file, the
- options specified in the Options resource will override
- the default options specified on the Include record.
-
- - Include records will be modified to permit referencing one or
- more Options resources. The Options will be used
- in the order listed on the Include record and the first
- one that matches will be applied.
-
- - Options (or specifications) currently supplied on the Include
- record will be deprecated (i.e. removed in a later version a
- year or so from now).
-
- - The Exclude record will be deprecated as the same functionality
- can be obtained by using an Exclude = yes in the Options.
-
-Options records:
- The following records can appear in the Options resource. An
- asterisk preceding the name indicates a feature not currently
- implemented.
-
- - Regexp "xxx" - Match regular expression
- - Wild "xxx" - Do a wild card match
-
- For Backup Jobs:
- - Compression= (GZIP, ...)
- - Signature= (MD5, SHA1, ...)
- - *Encryption=
- - OneFs= (yes/no) - remain on one filesystem
- - Recurse= (yes/no) - recurse into subdirectories
- - Sparse= (yes/no) - do sparse file backup
- - *Exclude= (yes/no) - exclude file from being saved
- - *Reader= (filename) - external read (backup) program
- - *Plugin= (filename) - read/write plugin module
-
- - Include= (yes/no) - Include the file matched no additional
- patterns are applied.
-
- For Verify Jobs:
- - verify= (ipnougsamc5) - verify options
-
- For Restore Jobs:
- - replace= (always/ifnewer/ifolder/never) - replace options currently
- implemented in 1.31
- - *Writer= (filename) - external write (restore) program
-
-
-Implementation:
- Currently options specifying compression, MD5 signatures, recursion,
- ... of a FileSet are supplied on the Include record. These will now
- all be collected into a Options resource, which will be
- specified in the Include in place of the options. Multiple Options
- may be specified. Since the Options may contain regular expressions
- that are applied to the full filename, this will give the ability
- to specify backup options on a file by file basis to whatever level
- of detail you wish.
-
-Example:
-
- Today:
-
- FileSet {
- Name = "FullSet"
- Include = compression=GZIP signature=MD5 {
- /
- }
- }
-
- Proposal:
-
- FileSet {
- Name = "FullSet"
- Include {
- Compression = GZIP;
- Signature = MD5
- Wild = /*.?*/ # matches all files.
- File = /
- }
- }
-
- That's a lot more to do the same thing, but it gives the ability to
- apply options on a file by file basis. For example, suppose you
- want to compress all files but not any file with extensions .gz or .Z.
- In that case, you will need to group two sets of options using
- the Options resource. Files may be anywhere except in an
- option set??? All OptionSets apply to all files in the order
- the OptionSets were specified. To have files included with
- different option sets without using wild-cards, use two or more
- Includes -- each one is handled in turn using only the files and
- optionsets specified in the include.
-
- FileSet {
- Name = "FullSet"
- Include {
- OptionSet {
- Signature = MD5
- # Note multiple Matches are ORed
- Wild = "*.gz" # matches .gz files
- Wild = "*.Z" # matches .Z files
- }
- OptionSet {
- Compression = GZIP
- Signature = MD5
- Wild = "*.?*" # matches all files
- }
- File = /
- }
- }
-
- Now, since the no Compression option is specified in the
- first group of Options, *.gz or *.Z file will have an MD5 signature computed,
- but will not be compressed. For all other files, the *.gz *.Z will not
- match, so the second group of options will be used which will include GZIP
- compression.
-
-Questions:
- - Is it necessary to provide some means of ANDing regular expressions
- and negation? (not currently planned)
-
- e.g. Wild = /*.gz/ && !/big.gz/
-
- - I see that Networker has a "null" module which, if specified, does not
- backup the file, but does make an record of the file in the catalog
- so that the catalog will reflect an exact picture of the filesystem.
- The result is that the file can be "seen" when "browsing" the save
- sets, but it cannot be restored.
-
- Is this really useful? Should it be implemented in Bacula?
-
-Results:
- After implementing the above, the user will be able to specify
- on a file by file basis (using regular expressions) what options are
- applied for the backup.
-
-
-=============================================
==========================================================
Unsaved File design
====
-=== Done in 1.33
-- Change console to bconsole.
-- Change smtp to bsmtp.
-- Fix time difference problem between Bacula and Client
- so that everything is in GMT.
-- Fix TimeZone problem!
-- Mount a tape that is not right for the job (wrong # files on tape)
- Bacula asks for another tape, fix problems with first tape and
- say "mount". All works OK, but status shows:
- Device /dev/nst0 open but no Bacula volume is mounted.
- Total Bytes=1,153,820,213 Blocks=17,888 Bytes/block=64,502
- Positioned at File=9 Block=3,951
- Full Backup job Rufus.2003-10-26_16.45.31 using Volume "DLT-24Oct03" on device /dev/nst0
- Files=21,003 Bytes=253,954,408 Bytes/sec=2,919,016
- FDReadSeqNo=192,134 in_msg=129830 out_msg=5 fd=7
-- Upgrade to cygwin 1.5
-- Optimize fsf not to read.
-- Use ioctl() fsf if it exists. Figure out where we are from
- the mt_status command. Use slow fsf only if other does not work.
-- Enhance "update slots" to include a "scan" feature
- scan 1; scan 1-5; scan 1,2,4 ... to update the catalog
-- Allow a slot or range of slots on the label barcodes command.
-- Finish implementation of Verify=DiskToCatalog
-- Make sure that Volumes are recycled based on "Least recently used"
- rather than lowest MediaId.
-- Add flag to write only one EOF mark on the tape.
-- Implement autochanger testing in btape "test" command.
-- Implement lmark to list everyfile marked.
-- Make mark/unmark report how many files marked/unmarked.
-- Keep last 5 or 10 completed jobs and show them in a similar list.
-- Make a Running Jobs: output similar to current Scheduled Jobs:
-- Change "create_media_record in bscan to use Archive instead of Full.
-- Have some way to estimate the restore size or have it printed.
-- Volume problems occurs if you have valid volume, written, then it is
- truncated. You get 12-Nov-2003 11:48 rufus-sd: kernsave.2003-11-12_11.48.09 Warning: mount.c:228 Volume on /tmp is not a Bacula labeled Volume, because:
- block.c:640 Read zero bytes on device /tmp.
-- Make sure that 64 bit I/O packets are used on Cygwin.
-- Add to supported autochangers
- OS : FreeBSD-4.9
- Auto-Changer : QUALSTAR TLS-4210
- Manufufactur : Qualstar
- Tapes : 12 (AIT1: 36GB, AIT2: 50GB all uncompressed)
- Drives : 2xAIT2 (installed in the Qualstar: SONY SDX-500C AIT2)
-- Document estimate command in tree.
-- Document lsmark command in tree.
-- Setup a standard job that builds a bootstrap file and saves
- it with the catalog database.
-- See if a restore job can add a file to the tape (prohibit this).
-- Restrict characters permitted in a name.
-- In restore, provide option for limiting to a particular Pool.
-- In restore, list FileSets that only have different base names --
- i.e. any FileSet with the same name should be treated as the same.
-- Make Scheduler sort jobs by StartTime, Priority.
-- Make sure smtp and any other useful program is executable by the world
- in case Bacula is not running as root.
-- Look at Dan's field width problems in PostgreSQL.
-- Look at effect of removing GROUP BYs.
-- In restore take all filesets with same base name.
-- From Alan Brown <ajb2@mssl.ucl.ac.uk>
- BTW, there's a make install bug in 1.33 - with --enable-gnome,
- gnome-console is built, but the binary and .conf are not being installed.
-- Permit Bacula and apcupsd donations (not done for apcupsd).
-- Fix Ctl-C crashing the Console (readline?).
-- Look at code in recycle_oldes_purged_volume() recycle.c. Why not
- let SQL do ORDER BY LastWritten ASC?
-- Look at find_next_volume() algorithm. Currently, it selects:
- +---------+------------+---------------------+-----------+
- | MediaId | VolumeName | LastWritten | VolBytes |
- +---------+------------+---------------------+-----------+
- | 3 | Test13 | 0000-00-00 00:00:00 | 1 |
- | 4 | Test14 | 0000-00-00 00:00:00 | 1 |
- | 1 | test11 | 2003-12-03 18:39:55 | 4,004,926 |
- | 2 | test12 | 2004-01-04 15:25:56 | 2,078,691 |
- +---------+------------+---------------------+-----------+
- but perhaps it should fill already used Volumes first, and use
- Append volumes before Purged, or Recycled, ...
-- Possibly remove the "|| ap == NULL" on lines 123 and 207 of lib/var.c,
- which creates compile problems on alpha systems.
- var.c:123: no match for `va_list & == long int'
-- Check "restore" 3 (JobId), then it asks for Storage resource. Does
- it verify that the correct volume is chosen?
-- Make Bacula "poll a drive".
-- Notes for final checking of Nic's code:
- Could I get you to double check the switch () statements in the
- job_check_maxwaittime and job_check_maxruntime functions in
- src/dird/job.c?
-- Define week of year for scheduler. W01, W02, ...
- Week 01 of a year is per definition the first week that has the
- Thursday in this year, which is equivalent to the week that contains the
- fourth day of January. In other words, the first week of a new year is
- the week that has the majority of its days in the new year. Week 01
- might also contain days from the previous year and the week before week
- 01 of a year is the last week (52 or 53) of the previous year even if it
- contains days from the new year. A week starts with Monday (day 1) and
- ends with Sunday (day 7). For example, the first week of the year 1997
- lasts from 1996-12-30 to 1997-01-05 and can be written in standard
- notation as
- 1997-W01 or 1997W01
- The week notation can also be extended by a number indicating the day
- of the week. For example, the day 1996-12-31, which is the Tuesday (day
- 2) of the first week of 1997, can also be written as
- 1997-W01-2 or 1997W012
-- Either restrict the characters in a name, or fix the problem
- emailing with names containing / (smtp command line breaks).
-- Implement .consolerc for Console
-- Implement scan: for every slot it finds, zero the slot of
- Volume other volume having that slot.
-- Make restore job check if all the files are actually restored.
-- Look at 2Gb limit for SQLite.
-- Fix get_storage_from_media_type (ua_restore) to use command line
- storage=
-- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume.
-- Write a mini-readline with history and editing.
-- Take a careful look a the Basic recycling algorithm. When Bacula
- chooses, the order should be:
- - Look for Append
- - Look for Recycle or Purged
- - Prune volumes
- - Look for purged
- Instead of using lowest media Id, find the least recently used
- volume.
-
- When the tape is mounted and Bacula requests the status
- - Do everything possible to use it.
-
- Define a "available" status, which is the currently mounted
- Volume and all volumes that are currently in the autochanger.
-- Is a pool specification really needed for a restore? Yes, and
- you may want to exclude archive Pools.
-- Implement a PostgreSQL driver.
-- Fix restore to list errors if Invalid block found, and if # files
- restored does not match # expected.
-- Something is not right in last block of fill command.
-- Add FileSet to command line arguments for restore.
-- Enhance time and size scanning routines.
-- Add Console usr permissions -- do by adding filters for
- jobs, clients, storage, ...
-- Put max network buffer size on a directive.
-- Why does "mark cygwin" take so long!!!!!!!!
-- Implement alist processing for ACLs from Console.
-- When a file is set for restore, walk back up the chain of
- directories, setting them to be restored.
-- Figure out a way to set restore on a directory without recursively
- decending. (recurse off?).
-- Fix restore to only pull in last Differential and later Incrementals.
-- Implement 3 Pools for a Job:
- Job {
- Name = ...
- Full Backup Pool = xxx
- Incremental Backup Pool = yyy
- Differential Backup Pool = zzz
- }
-- Look at ASSERT() at 384 src/lib/bnet.c
-- Dates are wrong in restore list from Win32 FD.
-- Dates are wrong in catalog from Win32 FD.
-- Remove h_errno from bnet.c by including proper header.
-- For "list jobs" order by EndTime.
-- Make two tape fill test work.
-- Add atime preservation.
-- Do not err job if could not write bootstrap file.
-- Save and restore last_job across executions.
-- Have each daemon save the last_jobs structure when exiting and
- read it back in when starting up.
-- "restore jobid=1 select" calls get_storage_xxx, which prints "JobId 1 is
- not running."
-- Make column listing for running jobs JobId Level Type Started Name Status
-- Why does Bacula need the drive open to do "autochanger list" ?
-- Add data compare on write/read in btape "test".
-- Rescue builds incorrect script files on Rufus.
-- Release SQLite 2.8.9
-- During install, copy any console.conf to bconsole.conf.
-- Check: Run = Level=Differential feb-dec 1 at 1:05 to see if wday is empty.
-- Look at installation file permissions with Scott so that make install
- and the rpms agree.
-- Finish code passing files=nnn to restore start.
-- Add ctl-c to console to stop current command and discard buffered
- output.
-- Estimate to Tibs never returns.
-- Symbolic link a directory to another one, then backup the symbolic link.
-- Check and possibly fix problems with hard links.
-- Fix query buffer clobber ua_query.c
-- Allow "delete job jobid=xx jobid=xxx".
-- Update volume FromPool (or FromPool=xxx) refreshes the Volume defaults from Pool.
-- Implement multiple Volume in "purge jobs volume=".
-- Test Qmsg() code to be used in bnet.c to prevent recursion. Queue the
- message. If dequeueing toss the messages. Lock while dequeuing so that
- it cannot be called recursively and set dequeuing flag.
-- On unknown client in restore "client=xxx"
- Could not find Client "Matou": ERR=Query failed: DROP TABLE temp1:
- ERR=no such table: temp1
-- Null dlist head and tail pointers if all items are removed.
-- Build console in client-only build.
-- Phil says that Windows file sizes mismatch in Verify when they should,
- and that either the file size or the catalog size was zero.
-- Check time/dates printed during restore when using Win32 API.
-- Once a job is canceled, make sure the status reflects that even if the
- job is "stuck" in the run queue.
-- Test work on conio.c -- particularly linking.
-- Complete Win32 installer
-- Add ms disk example
+=== Done in 1.34.3
+- To pass Include 1 or two letter commands
+ I Name Include name - first record
+ B Name Base name - repeat
+ R "xxx" Regexp
+ W "xxx" Wild Card
+ E zzz Exclude expression (wild card)
+ P "plugin" Plugin
+ D "reader" Reader program
+ T "writer" Writer program
+ O Options In current commpressed format
+ (compression, signature, onefs, recurse, sparse,
+ replace, verify options, ...)
+ N End option set
+ B BaseName Start second option set any letter
+ ...
+ E
+ F Number Number of filenames to follow
+ <file-name>
+ B Name
+ ...
+ N End option set
+ F Number Number of filenames to follow
+ <file-name>
+ ...
+- Add regression of btape "fill"
+==== Done in 1.34.4
+- Implement "label slots=1,2... pool=yyy barcodes"
+- Mark a non-removable disk Volume in error rather than asking sys op to
+ mount it.
+- Implement Fixed storage LabelFormat test.
+- Add reporting in attr despooling.
+- Figure out how to use ssh or stunnel to protect Bacula communications.
+ Add Dan's work to manual See ssl.txt
+
+==== Done in 1.35.0
+- Implement fast tree insert (doubly linked list)
+- Disallow using Internal database
+- Feedback while the restore tree is being built.
+- Add detection of Whitebox release in ./configure
+- Add InChanger to "list media"
+- Implement second cut of SIGHUP in Dir
+ (Implement resources on a single pointer)
+- Add regular expressions to FileSets (needs testing).
+- Test new despooling error recovery code when partition fills.
+- Sort Scheduled jobs status listing by start time.
+- Add priority to Scheduled jobs status listing.
+- Look at Win32 inc problem when new directory added files not saved???
+- Require resource names to be unique.
+- Make sure that all errors from libacl are printed.
+- Make FD run on Win95 if possible (not really possible).
+- Fix GROUP BYs in all SQL (done by Dan -- thanks)
+- Make entering multiple Storage names in Dir illegal.
+- Make btape release the drive during the "test" append.
+- Fix restore:
+ You have selected the following JobIds: 739,856,797
+ Building directory tree for JobId 739 ...
+ Building directory tree for JobId 856 ...
+ Building directory tree for JobId 797 ...3
+ Jobs inserted into the tree.
+- Fix errno handling in win32 compat routines.
+- Add better error codes to run_program (10000+)
+- Revisit and revise Disaster Recovery (fix SCSI and RAID
+ disk detection)
+