X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fkernstodo;h=64c5a3105f035587012f54c513b83eb5a7865c6f;hb=825f8335ea465486c42e2d1f6c10d7310d25dc9a;hp=2571adee1d89ac9a808ce2f03c819f1e36b1e6df;hpb=e22b58a14b3900c5a47315908cb15ac55243fd63;p=bacula%2Fbacula diff --git a/bacula/kernstodo b/bacula/kernstodo index 2571adee1d..64c5a3105f 100644 --- a/bacula/kernstodo +++ b/bacula/kernstodo @@ -1,81 +1,209 @@ Kern's ToDo List - 16 June 2005 + 20 July 2005 Major development: Project Developer ======= ========= -TLS Landon Fuller -Unicode in Win32 Thorsten Engel (done) -VSS Thorsten Engel (in beta testing) Version 1.37 Kern (see below) ======================================================== -1.37 Major Projects: -#3 Migration (Move, Copy, Archive Jobs) - (probably not this version) -#7 Single Job Writing to Multiple Storage Devices - (probably not this version) - -## Create a new GUI chapter explaining all the GUI programs. - -Autochangers: -- Make "update slots" when pointing to Autochanger, remove - all Volumes from other drives. "update slots all-drives"? +Final items for 1.37 before release: +1. Fix bugs +- --without-openssl breaks at least on Solaris. +3. Document all the new features (about half done). + - VSS. Shall I write "Include Writer 'WMI Writer", "MSDEWriter" + Let me explain this: An windows application can (no must) + register as VSS writer. This means that the applications opts to + be notified if a backup (or restore) occurs. If it then gets + this message, it will immediately store a consistent state to + disk. Examples for these writers are "MSDE" (Microsoft database + engine), "Event Log Writer", "Registry Writer" plus 3rd + party-writers. If you have a non-vss aware application (e.g. + SQL Anywhere or probably MySQL), a shadow copy is still generated + and the open files can be backed up, but there is no guarantee + that the file is consistent. + + At least the Microsoft example makes a significant effort to + determine which writers may be involved when a drive or file is + to be shadow copied. Of course, every single writer may fail.... + So they offer a user interface to explicitly include or exclude a + writer when creating a VSS shadow copy. I personally would not + like to bother the user with this - at least not right now + (efforts for exchanging lists between fd and director + efforts + for selecting, etc.). + + But I personally would like to have an information message about + the individual writers involved in the backup-process ("vssadmin + list writers" produced 4 entries on my xp-client and 7 on my w2k3 + server, please try this on your machine to understand the system + a little bit better). + + - Multiple drive autochanger support + - Support for ANSI/IBM labels. + - Seven new options keywords in a FileSet resource: + ignorecase, fstype, hfsplussupport, wilddir, wildfile, regexdir, + and regexfile thanks to Pruben Guldberg). See below for details. + - Restore of all files for a Job or set of jobs even if the file + records have been removed from the catalog. + - Restore of a directory (non-recursive, i.e. only one level). + - Support for TLS (ssl) between all the daemon connections thanks + to Landon Fuller. + - Any Volume in the Pool named Scratch may be reassigned to any + other Pool when a new Volume is needed. + - Unicode filename support for Win32 (thanks to Thorsten Engel) + - Volume Shadow Copy support for Win32 thus the capability to + backup exclusively opened files (thanks to Thorsten Engel). + A VSS enabled Win32 FD is available. You must explicitly + turn on VSS with "Enable VSS = yes" in your FileSet resource. + - SQLite3 support, but it seems to run at 1/2 to 1/4 the speed of + SQLite2. + - New Job directive "Prefer Mounted Volumes = yes|no" causes the + SD to select either an Autochanger or a drive with a valid + Volume already mounted in preference. If none is available, + it will select the first available drive. + - New Run directive in Job resource of DIR. It permits + cloning of jobs. To clone a copy of the current job, use + Run = "job-name level=%l since=\"%s\"" + Note, job-name is normally the same name as the job that + is running but there is no restriction on what you put. If you + want to start the job by hand and use job overrides such as + storage=xxx, realize that the job will be started with the + default storage values not the overrides. The level=%l guarantees + that the chosen level of the job is the same, and the since=... + ensures that the job uses *exactly* the same time/date for incremental + and differential jobs. The since=... is ignored when level=Full. + A cloned job will not start additional clones, so it is not possible + to recurse. + - New Options keywords in a FileSet directive: + - WildDir xxx + Will do a wild card match against directories (files will not + be matched). + - WildFile xxx + Will do a wild card match against files (directories will not + be matched). + - RegexDir xxx + Will do a regular expression match against directories (files + will not be matched). + - RegexFile xxx + Will do a regular expression match against files( directories + will not be matched). + - IgnoreCase = yes | no + Will ignore case in wild card and regular expression matches. + This is handy for Windows where filename case is not significant. + - FsType = string + where string is a filesystem type: ext2, jfs, ntfs, proc, + reiserfs, xfs, usbdevfs, sysfs, smbfs, iso9660. For ext3 + systems, use ext2. You may have multiple fstype directives + and thus permit multiple filesystem types. If the type + specified on the fstype directive does not match the + filesystem for a particular directive, that directory will + not be backed up. This directive can be used to prevent + backing up non-local filesystems. + - HFS Plus Support = yes | no + If set, Mac OS X resource forks will be saved and restored. + - Label Type = ANSI | IBM | Bacula + Implemented in Director Pool resource and in SD Device resource. + If it is specified in the SD Device resource, it will take + precedence over the value passed from the Director to the SD. + - Check Labels = yes | no + Implemented in the SD Device resource. If you intend to read + ANSI or IBM labels, this *must* be set. Even if the volume + is not ANSI labeled, you can set this to yes, and Bacula will + check the label type. + - Scripts Directory = name. Defines the directory from + which Bacula scripts will be called for events. In fact, Bacula + appends this name to the standard Python list of search directories, + so the script could also be in any of the Python system directories. + - In FileSet, you can exclude backing up of hardlinks (if you have + a lot, it can be very expensive), by using: + HardLinks = no + in the Options section. Patch supplied by David R Bosso. Thanks. + - MaximumPartSize = bytes (SD, Device resource) + Defines the maximum part size. + - Requires Mount = Yes/No (SD, Device resource) + Defines if the device require to be mounted to be read, and if it + must be written in a special way. If it set, the following directives + must be defined in the same Device resource: + + Mount Point = directory + Directory where the device must be mounted. + + Mount Command = name-string + Command that must be executed to mount the device. Before the command + is executed, %a is replaced with the Archive Device, and %m with the + Mount Point. + + Unmount Command = name-string + Command that must be executed to unmount the device. Before the + command is executed, %a is replaced with the Archive Device, and + %m with the Mount Point. + + Write Part Command = name-string + Command that must be executed to write a part to the device. Before + the command is executed, %a is replaced with the Archive Device, %m + with the Mount Point, %n with the current part number (0-based), + and %v with the current part filename. + + Free Space Command = name-string + Command that must be executed to check how much free space is left + on the device. Before the command is executed, %a is replaced with + the Archive Device, %m with the Mount Point, %n with the current part + number (0-based), and %v with the current part filename. + - Write Part After Job = Yes/No (DIR, Job Resource, and Schedule Resource) + If this directive is set to yes (default no), a new part file will be + created after the job is finished. + - A pile of new Directives to support TLS. Please see the TLS chapter + of the manual. + + - "python restart" restarts the Python interpreter. Rather brutal, make + sure no Python scripts are running. This permits you to change + a Python script and ge + - With Python 2.3, there are a few compiler warnings. + - You must add --with-openssl to the configure command line if + you want TLS communications encryption support. +7. Write a bacula-web document +9. Run the regression scripts on Solaris and FreeBSD +- Figure out how to package gui, and rescue programs. +- Test TLS. Document: -- Port limiting -m in iptables to prevent DoS attacks - could cause broken pipes on Bacula. -- Document that Bootstrap files can be written with cataloging - turned off. -- Pruning with Admin job. -- Add better documentation on how restores can be done -- OS linux 2.4 - 1) ADIC, DLT, FastStor 4000, 7*20GB - 2) Sun, DDS, (Suns name unknown - Archive Python DDS drive), 1.2GB - 3) Wangtek, QIC, 6525ES, 525MB (fixed block size 1k, block size etc. - driver dependent - aic7xxx works, ncr53c8xx with problems) - 4) HP, DDS-2, C1553A, 6*4GB -- Doc the following - to activate, check or disable the hardware compression feature on my - exb-8900 i use the exabyte "MammothTool" you can get it here: - http://www.exabyte.com/support/online/downloads/index.cfm - There is a solaris version of this tool. With option -C 0 or 1 you can - disable or activate compression. Start this tool without any options for - a small reference. -- Linux Sony LIB-D81, AIT-3 library works. -- Document PostgreSQL performance problems bug 131. -- Document testing -- Document that ChangerDevice is used for Alert command. -- Document new CDROM directory. -- Document Heartbeat Interval in the dealing with firewalls section. +- Document cleaning up the spool files: + db, pid, state, bsr, mail, conmsg, spool - Document the multiple-drive-changer.txt script. +- Pruning with Admin job. -For 1.37: -- Cleaning tapes should have Status "Cleaning" rather than append. + +For 1.39: +- Fix bpipe.c so that it does not modify results pointer. + ***FIXME*** calling sequence should be changed. +1.xx Major Projects: +#3 Migration (Move, Copy, Archive Jobs) +#7 Single Job Writing to Multiple Storage Devices +- Reserve blocks other restore jobs when first cannot connect + to SD. +- Add true/false to conf same as yes/no +- For Windows disaster recovery see http://unattended.sf.net/ +- regardless of the retention period, Bacula will not prune the + last Full, Diff, or Inc File data until a month after the + retention period for the last Full backup that was done. +- update volume=xxx --- add status=Full +- Remove old spool files on startup. +- Exclude SD spool/working directory. - Refuse to prune last valid Full backup. Same goes for Catalog. -- Why is SpoolDirectory = /home/bacula/spool; not reported - as an error when writing a DVD? -- Add setting Volume State via Python. -- Make bootstrap file handle multiple MediaTypes (SD) -- --without-openssl breaks at least on Solaris. - Python: - Make a callback when Rerun failed levels is called. - Give Python program access to Scheduled jobs. + - Add setting Volume State via Python. - Python script to save with Python, not save, save with Bacula. - Python script to do backup. - What events? - Change the Priority, Client, Storage, JobStatus (error) at the start of a job. - - Make sure that Python has access to Client address/port so that - it can check if Clients are alive. - +- Why is SpoolDirectory = /home/bacula/spool; not reported + as an error when writing a DVD? +- Make bootstrap file handle multiple MediaTypes (SD) - Remove all old Device resource code in Dir and code to pass it back in SD -- better, rework it to pass back device statistics. - Check locking of resources -- be sure to lock devices where previously resources were locked. -- Add global lock on all devices when creating a device structure. +- The last part is left in the spool dir. + -Maybe in 1.37: - In restore don't compare byte count on a raw device -- directory entry does not contain bytes. - To mark files as deleted, run essentially a Verify to disk, and @@ -130,6 +258,61 @@ Maybe in 1.37: - Bug: if a job is manually scheduled to run later, it does not appear in any status report and cannot be cancelled. +==== Keeping track of deleted files ==== + My "trick" for keeping track of deletions is the following. + Assuming the user turns on this option, after all the files + have been backed up, but before the job has terminated, the + FD will make a pass through all the files and send their + names to the DIR (*exactly* the same as what a Verify job + currently does). This will probably be done at the same + time the files are being sent to the SD avoiding a second + pass. The DIR will then compare that to what is stored in + the catalog. Any files in the catalog but not in what the + FD sent will receive a catalog File entry that indicates + that at that point in time the file was deleted. + + During a restore, any file initially picked up by some + backup (Full, ...) then subsequently having a File entry + marked "delete" will be removed from the tree, so will not + be restored. If a file with the same name is later OK it + will be inserted in the tree -- this already happens. All + will be consistent except for possible changes during the + running of the FD. + + Since I'm on the subject, some of you may be wondering what + the utility of the in memory tree is if you are going to + restore everything (at least it comes up from time to time + on the list). Well, it is still *very* useful because it + allows only the last item found for a particular filename + (full path) to be entered into the tree, and thus if a file + is backed up 10 times, only the last copy will be restored. + I recently (last Friday) restored a complete directory, and + the Full and all the Differential and Incremental backups + spanned 3 Volumes. The first Volume was not even mounted + because all the files had been updated and hence backed up + since the Full backup was made. In this case, the tree + saved me a *lot* of time. + + Make sure this information is stored on the tape too so + that it can be restored directly from the tape. + + Comments from Martin Simmons (I think they are all covered): + Ok, that should cover the basics. There are few issues though: + + - Restore will depend on the catalog. I think it is better to include the + extra data in the backup as well, so it can be seen by bscan and bextract. + + - I'm not sure if it will preserve multiple hard links to the same inode. Or + maybe adding or removing links will cause the data to be dumped again? + + - I'm not sure if it will handle renamed directories. Possibly it will work + by dumping the whole tree under a renamed directory? + + - It remains to be seen how the backup performance of the DIR's will be + affected when comparing the catalog for a large filesystem. + +===== + Regression tests: - Add Pool/Storage override regression test. - Add delete JobId to regression. @@ -1294,3 +1477,51 @@ Block Position: 0 - Implement "PreferMountedVolumes = yes|no" in Job resource. ## Integrate web-bacula into a new Bacula project with bimagemgr. +- Cleaning tapes should have Status "Cleaning" rather than append. +- Make sure that Python has access to Client address/port so that + it can check if Clients are alive. +- Review all items in "restore". +- Fix PostgreSQL GROUP BY problems in restore. +- Fix PostgreSQL sql problems in bugs. +- After rename + 04-Jul 13:01 MainSD: Rufus.2005-07-04_01.05.02 Warning: Director wanted Volume + "DLT-13Feb04". + Current Volume "DLT-04Jul05" not acceptable because: + 1997 Volume "DLT-13Feb04" not in catalog. + 04-Jul 13:01 MainSD: Please mount Volume "DLT-04Jul05" on Storage Device + "HP DLT 80" (/dev/nst0) for Job Rufus.2005-07-04_01.05.02 +## Create a new GUI chapter explaining all the GUI programs. +- Make "update slots" when pointing to Autochanger, remove + all Volumes from other drives. "update slots all-drives"? + No, this is done by modifying mtx-changer to list what is + in the drives. +- Finish TLS implementation. +- Port limiting -m in iptables to prevent DoS attacks + could cause broken pipes on Bacula. +6. Build and test the Volume Shadow Copy (VSS) for Win32. +- Allow cancel of unknown Job +- State not saved when closing Win32 FD by icon +- bsr-opt-test fails. bsr deleted. Fix. +- Move Python daemon variables from Job to Bacula object. + WorkingDir, ConfigFile +- Document that Bootstrap files can be written with cataloging + turned off. +- Document details of ANSI/IBM labels +- OS linux 2.4 + 1) ADIC, DLT, FastStor 4000, 7*20GB +- Linux Sony LIB-D81, AIT-3 library works. +- Doc the following + to activate, check or disable the hardware compression feature on my + exb-8900 i use the exabyte "MammothTool" you can get it here: + http://www.exabyte.com/support/online/downloads/index.cfm + There is a solaris version of this tool. With option -C 0 or 1 you can + disable or activate compression. Start this tool without any options for + a small reference. +- Document Heartbeat Interval in the dealing with firewalls section. +- Document new CDROM directory. +- On Win32 working directory must have drive letter ???? +- On Win32 working directory must be writable by SYSTEM to + do restores. +- Document that ChangerDevice is used for Alert command. +- Add better documentation on how restores can be done +8. Take one more try at making DVD writing work (no go)