Kern's ToDo List
- 16 June 2005
+ 20 July 2005
Major development:
Project Developer
======= =========
-TLS Landon Fuller
-Unicode in Win32 Thorsten Engel (done)
-VSS Thorsten Engel (in beta testing)
Version 1.37 Kern (see below)
========================================================
-1.37 Major Projects:
-#3 Migration (Move, Copy, Archive Jobs)
- (probably not this version)
-#7 Single Job Writing to Multiple Storage Devices
- (probably not this version)
-
-## Create a new GUI chapter explaining all the GUI programs.
-
-Autochangers:
-- Make "update slots" when pointing to Autochanger, remove
- all Volumes from other drives. "update slots all-drives"?
+Final items for 1.37 before release:
+1. Fix bugs
+- --without-openssl breaks at least on Solaris.
+3. Document all the new features (about half done).
+ - VSS. Shall I write "Include Writer 'WMI Writer", "MSDEWriter"
+ Let me explain this: An windows application can (no must)
+ register as VSS writer. This means that the applications opts to
+ be notified if a backup (or restore) occurs. If it then gets
+ this message, it will immediately store a consistent state to
+ disk. Examples for these writers are "MSDE" (Microsoft database
+ engine), "Event Log Writer", "Registry Writer" plus 3rd
+ party-writers. If you have a non-vss aware application (e.g.
+ SQL Anywhere or probably MySQL), a shadow copy is still generated
+ and the open files can be backed up, but there is no guarantee
+ that the file is consistent.
+
+ At least the Microsoft example makes a significant effort to
+ determine which writers may be involved when a drive or file is
+ to be shadow copied. Of course, every single writer may fail....
+ So they offer a user interface to explicitly include or exclude a
+ writer when creating a VSS shadow copy. I personally would not
+ like to bother the user with this - at least not right now
+ (efforts for exchanging lists between fd and director + efforts
+ for selecting, etc.).
+
+ But I personally would like to have an information message about
+ the individual writers involved in the backup-process ("vssadmin
+ list writers" produced 4 entries on my xp-client and 7 on my w2k3
+ server, please try this on your machine to understand the system
+ a little bit better).
+
+ - Multiple drive autochanger support
+ - Support for ANSI/IBM labels.
+ - Seven new options keywords in a FileSet resource:
+ ignorecase, fstype, hfsplussupport, wilddir, wildfile, regexdir,
+ and regexfile thanks to Pruben Guldberg). See below for details.
+ - Restore of all files for a Job or set of jobs even if the file
+ records have been removed from the catalog.
+ - Restore of a directory (non-recursive, i.e. only one level).
+ - Support for TLS (ssl) between all the daemon connections thanks
+ to Landon Fuller.
+ - Any Volume in the Pool named Scratch may be reassigned to any
+ other Pool when a new Volume is needed.
+ - Unicode filename support for Win32 (thanks to Thorsten Engel)
+ - Volume Shadow Copy support for Win32 thus the capability to
+ backup exclusively opened files (thanks to Thorsten Engel).
+ A VSS enabled Win32 FD is available. You must explicitly
+ turn on VSS with "Enable VSS = yes" in your FileSet resource.
+ - SQLite3 support, but it seems to run at 1/2 to 1/4 the speed of
+ SQLite2.
+ - New Job directive "Prefer Mounted Volumes = yes|no" causes the
+ SD to select either an Autochanger or a drive with a valid
+ Volume already mounted in preference. If none is available,
+ it will select the first available drive.
+ - New Run directive in Job resource of DIR. It permits
+ cloning of jobs. To clone a copy of the current job, use
+ Run = "job-name level=%l since=\"%s\""
+ Note, job-name is normally the same name as the job that
+ is running but there is no restriction on what you put. If you
+ want to start the job by hand and use job overrides such as
+ storage=xxx, realize that the job will be started with the
+ default storage values not the overrides. The level=%l guarantees
+ that the chosen level of the job is the same, and the since=...
+ ensures that the job uses *exactly* the same time/date for incremental
+ and differential jobs. The since=... is ignored when level=Full.
+ A cloned job will not start additional clones, so it is not possible
+ to recurse.
+ - New Options keywords in a FileSet directive:
+ - WildDir xxx
+ Will do a wild card match against directories (files will not
+ be matched).
+ - WildFile xxx
+ Will do a wild card match against files (directories will not
+ be matched).
+ - RegexDir xxx
+ Will do a regular expression match against directories (files
+ will not be matched).
+ - RegexFile xxx
+ Will do a regular expression match against files( directories
+ will not be matched).
+ - IgnoreCase = yes | no
+ Will ignore case in wild card and regular expression matches.
+ This is handy for Windows where filename case is not significant.
+ - FsType = string
+ where string is a filesystem type: ext2, jfs, ntfs, proc,
+ reiserfs, xfs, usbdevfs, sysfs, smbfs, iso9660. For ext3
+ systems, use ext2. You may have multiple fstype directives
+ and thus permit multiple filesystem types. If the type
+ specified on the fstype directive does not match the
+ filesystem for a particular directive, that directory will
+ not be backed up. This directive can be used to prevent
+ backing up non-local filesystems.
+ - HFS Plus Support = yes | no
+ If set, Mac OS X resource forks will be saved and restored.
+ - Label Type = ANSI | IBM | Bacula
+ Implemented in Director Pool resource and in SD Device resource.
+ If it is specified in the SD Device resource, it will take
+ precedence over the value passed from the Director to the SD.
+ - Check Labels = yes | no
+ Implemented in the SD Device resource. If you intend to read
+ ANSI or IBM labels, this *must* be set. Even if the volume
+ is not ANSI labeled, you can set this to yes, and Bacula will
+ check the label type.
+ - Scripts Directory = <directory> name. Defines the directory from
+ which Bacula scripts will be called for events. In fact, Bacula
+ appends this name to the standard Python list of search directories,
+ so the script could also be in any of the Python system directories.
+ - In FileSet, you can exclude backing up of hardlinks (if you have
+ a lot, it can be very expensive), by using:
+ HardLinks = no
+ in the Options section. Patch supplied by David R Bosso. Thanks.
+ - MaximumPartSize = bytes (SD, Device resource)
+ Defines the maximum part size.
+ - Requires Mount = Yes/No (SD, Device resource)
+ Defines if the device require to be mounted to be read, and if it
+ must be written in a special way. If it set, the following directives
+ must be defined in the same Device resource:
+ + Mount Point = directory
+ Directory where the device must be mounted.
+ + Mount Command = name-string
+ Command that must be executed to mount the device. Before the command
+ is executed, %a is replaced with the Archive Device, and %m with the
+ Mount Point.
+ + Unmount Command = name-string
+ Command that must be executed to unmount the device. Before the
+ command is executed, %a is replaced with the Archive Device, and
+ %m with the Mount Point.
+ + Write Part Command = name-string
+ Command that must be executed to write a part to the device. Before
+ the command is executed, %a is replaced with the Archive Device, %m
+ with the Mount Point, %n with the current part number (0-based),
+ and %v with the current part filename.
+ + Free Space Command = name-string
+ Command that must be executed to check how much free space is left
+ on the device. Before the command is executed, %a is replaced with
+ the Archive Device, %m with the Mount Point, %n with the current part
+ number (0-based), and %v with the current part filename.
+ - Write Part After Job = Yes/No (DIR, Job Resource, and Schedule Resource)
+ If this directive is set to yes (default no), a new part file will be
+ created after the job is finished.
+ - A pile of new Directives to support TLS. Please see the TLS chapter
+ of the manual.
+
+ - "python restart" restarts the Python interpreter. Rather brutal, make
+ sure no Python scripts are running. This permits you to change
+ a Python script and ge
+ - With Python 2.3, there are a few compiler warnings.
+ - You must add --with-openssl to the configure command line if
+ you want TLS communications encryption support.
+7. Write a bacula-web document
+9. Run the regression scripts on Solaris and FreeBSD
+- Figure out how to package gui, and rescue programs.
+- Test TLS.
Document:
-- Document that Bootstrap files can be written with cataloging
- turned off.
-- Pruning with Admin job.
-- Add better documentation on how restores can be done
-- OS linux 2.4
- 1) ADIC, DLT, FastStor 4000, 7*20GB
- 2) Sun, DDS, (Suns name unknown - Archive Python DDS drive), 1.2GB
- 3) Wangtek, QIC, 6525ES, 525MB (fixed block size 1k, block size etc.
- driver dependent - aic7xxx works, ncr53c8xx with problems)
- 4) HP, DDS-2, C1553A, 6*4GB
-- Doc the following
- to activate, check or disable the hardware compression feature on my
- exb-8900 i use the exabyte "MammothTool" you can get it here:
- http://www.exabyte.com/support/online/downloads/index.cfm
- There is a solaris version of this tool. With option -C 0 or 1 you can
- disable or activate compression. Start this tool without any options for
- a small reference.
-- Linux Sony LIB-D81, AIT-3 library works.
-- Document PostgreSQL performance problems bug 131.
-- Document testing
-- Document that ChangerDevice is used for Alert command.
-- Document new CDROM directory.
-- Document Heartbeat Interval in the dealing with firewalls section.
+- Document cleaning up the spool files:
+ db, pid, state, bsr, mail, conmsg, spool
- Document the multiple-drive-changer.txt script.
+- Pruning with Admin job.
-For 1.37:
+
+For 1.39:
+- Fix bpipe.c so that it does not modify results pointer.
+ ***FIXME*** calling sequence should be changed.
+1.xx Major Projects:
+#3 Migration (Move, Copy, Archive Jobs)
+#7 Single Job Writing to Multiple Storage Devices
+- Reserve blocks other restore jobs when first cannot connect
+ to SD.
+- Add true/false to conf same as yes/no
+- For Windows disaster recovery see http://unattended.sf.net/
+- regardless of the retention period, Bacula will not prune the
+ last Full, Diff, or Inc File data until a month after the
+ retention period for the last Full backup that was done.
+- update volume=xxx --- add status=Full
+- Remove old spool files on startup.
+- Exclude SD spool/working directory.
- Refuse to prune last valid Full backup. Same goes for Catalog.
-- Why is SpoolDirectory = /home/bacula/spool; not reported
- as an error when writing a DVD?
-- Add setting Volume State via Python.
-- Make bootstrap file handle multiple MediaTypes (SD)
-- --without-openssl breaks at least on Solaris.
- Python:
- Make a callback when Rerun failed levels is called.
- Give Python program access to Scheduled jobs.
+ - Add setting Volume State via Python.
- Python script to save with Python, not save, save with Bacula.
- Python script to do backup.
- What events?
- Change the Priority, Client, Storage, JobStatus (error)
at the start of a job.
- - Make sure that Python has access to Client address/port so that
- it can check if Clients are alive.
-
+- Why is SpoolDirectory = /home/bacula/spool; not reported
+ as an error when writing a DVD?
+- Make bootstrap file handle multiple MediaTypes (SD)
- Remove all old Device resource code in Dir and code to pass it
back in SD -- better, rework it to pass back device statistics.
- Check locking of resources -- be sure to lock devices where previously
resources were locked.
-- Add global lock on all devices when creating a device structure.
+- The last part is left in the spool dir.
+
-Maybe in 1.37:
+- In restore don't compare byte count on a raw device -- directory
+ entry does not contain bytes.
- To mark files as deleted, run essentially a Verify to disk, and
when a file is found missing (MarkId != JobId), then create
a new File record with FileIndex == -1. This could be done
- Bug: if a job is manually scheduled to run later, it does not appear
in any status report and cannot be cancelled.
+==== Keeping track of deleted files ====
+ My "trick" for keeping track of deletions is the following.
+ Assuming the user turns on this option, after all the files
+ have been backed up, but before the job has terminated, the
+ FD will make a pass through all the files and send their
+ names to the DIR (*exactly* the same as what a Verify job
+ currently does). This will probably be done at the same
+ time the files are being sent to the SD avoiding a second
+ pass. The DIR will then compare that to what is stored in
+ the catalog. Any files in the catalog but not in what the
+ FD sent will receive a catalog File entry that indicates
+ that at that point in time the file was deleted.
+
+ During a restore, any file initially picked up by some
+ backup (Full, ...) then subsequently having a File entry
+ marked "delete" will be removed from the tree, so will not
+ be restored. If a file with the same name is later OK it
+ will be inserted in the tree -- this already happens. All
+ will be consistent except for possible changes during the
+ running of the FD.
+
+ Since I'm on the subject, some of you may be wondering what
+ the utility of the in memory tree is if you are going to
+ restore everything (at least it comes up from time to time
+ on the list). Well, it is still *very* useful because it
+ allows only the last item found for a particular filename
+ (full path) to be entered into the tree, and thus if a file
+ is backed up 10 times, only the last copy will be restored.
+ I recently (last Friday) restored a complete directory, and
+ the Full and all the Differential and Incremental backups
+ spanned 3 Volumes. The first Volume was not even mounted
+ because all the files had been updated and hence backed up
+ since the Full backup was made. In this case, the tree
+ saved me a *lot* of time.
+
+ Make sure this information is stored on the tape too so
+ that it can be restored directly from the tape.
+
+ Comments from Martin Simmons (I think they are all covered):
+ Ok, that should cover the basics. There are few issues though:
+
+ - Restore will depend on the catalog. I think it is better to include the
+ extra data in the backup as well, so it can be seen by bscan and bextract.
+
+ - I'm not sure if it will preserve multiple hard links to the same inode. Or
+ maybe adding or removing links will cause the data to be dumped again?
+
+ - I'm not sure if it will handle renamed directories. Possibly it will work
+ by dumping the whole tree under a renamed directory?
+
+ - It remains to be seen how the backup performance of the DIR's will be
+ affected when comparing the catalog for a large filesystem.
+
+=====
+
Regression tests:
- Add Pool/Storage override regression test.
- Add delete JobId to regression.
- Implement "PreferMountedVolumes = yes|no" in Job resource.
## Integrate web-bacula into a new Bacula project with
bimagemgr.
+- Cleaning tapes should have Status "Cleaning" rather than append.
+- Make sure that Python has access to Client address/port so that
+ it can check if Clients are alive.
+- Review all items in "restore".
+- Fix PostgreSQL GROUP BY problems in restore.
+- Fix PostgreSQL sql problems in bugs.
+- After rename
+ 04-Jul 13:01 MainSD: Rufus.2005-07-04_01.05.02 Warning: Director wanted Volume
+ "DLT-13Feb04".
+ Current Volume "DLT-04Jul05" not acceptable because:
+ 1997 Volume "DLT-13Feb04" not in catalog.
+ 04-Jul 13:01 MainSD: Please mount Volume "DLT-04Jul05" on Storage Device
+ "HP DLT 80" (/dev/nst0) for Job Rufus.2005-07-04_01.05.02
+## Create a new GUI chapter explaining all the GUI programs.
+- Make "update slots" when pointing to Autochanger, remove
+ all Volumes from other drives. "update slots all-drives"?
+ No, this is done by modifying mtx-changer to list what is
+ in the drives.
+- Finish TLS implementation.
+- Port limiting -m in iptables to prevent DoS attacks
+ could cause broken pipes on Bacula.
+6. Build and test the Volume Shadow Copy (VSS) for Win32.
+- Allow cancel of unknown Job
+- State not saved when closing Win32 FD by icon
+- bsr-opt-test fails. bsr deleted. Fix.
+- Move Python daemon variables from Job to Bacula object.
+ WorkingDir, ConfigFile
+- Document that Bootstrap files can be written with cataloging
+ turned off.
+- Document details of ANSI/IBM labels
+- OS linux 2.4
+ 1) ADIC, DLT, FastStor 4000, 7*20GB
+- Linux Sony LIB-D81, AIT-3 library works.
+- Doc the following
+ to activate, check or disable the hardware compression feature on my
+ exb-8900 i use the exabyte "MammothTool" you can get it here:
+ http://www.exabyte.com/support/online/downloads/index.cfm
+ There is a solaris version of this tool. With option -C 0 or 1 you can
+ disable or activate compression. Start this tool without any options for
+ a small reference.
+- Document Heartbeat Interval in the dealing with firewalls section.
+- Document new CDROM directory.
+- On Win32 working directory must have drive letter ????
+- On Win32 working directory must be writable by SYSTEM to
+ do restores.
+- Document that ChangerDevice is used for Alert command.
+- Add better documentation on how restores can be done
+8. Take one more try at making DVD writing work (no go)
projects / bacula / bacula / blobdiff