Kern's ToDo List
- 04 July 2005
+ 01 October 2005
Major development:
Project Developer
======= =========
-TLS Landon Fuller
-Unicode in Win32 Thorsten Engel (done)
-VSS Thorsten Engel (in beta testing)
Version 1.37 Kern (see below)
========================================================
-1.37 Major Projects:
+Final items for 1.37 before release:
+1. Fix bugs
+- Look at fixing restore status stats in SD.
+- Mount after manually unloading changer causes hang in SD
+- Close STDOUT if debug_level == 0
+- Check if ANSI tape labeling works with drive in
+ read-only mode.
+ > > btape: label.c:299 write_volume_label()
+ > > btape: label.c:302 Label type=0
+ > > btape: dev.c:648 rewind_dev fd=3 "VTS0" (/dev/tape0)
+ > > btape: label.c:530 Start create_volume_label()
+ > >
+ > > Volume Label:
+ > > Id : Bacula 1.0 immortal
+ > > VerNo : 11
+ > > VolName : 450340
+ > > PrevVolName :
+ > > VolFile : 0
+ > > LabelType : PRE_LABEL
+ > > LabelSize : 0
+ > > PoolName : Default
+ > > MediaType : VTS
+ > > PoolType : Backup
+ > > HostName : sysrmr.eia.doe.gov
+ > > btape: ansi_label.c:282 Write ANSI label type=2
+ > > 15-Sep 13:12 btape: btape Fatal error: ansi_label.c:303 Could not
+ > > write ANSI VOL1
+ > > label. ERR=Bad file descriptor
+
+- Check "update slots=7 scan storage=DLT drive=0" with
+ non-bacula tape in the drive.
+
+- --without-openssl breaks at least on Solaris.
+- Figure out how to package gui, and rescue programs.
+- Test TLS.
+- Arno had to do -- to get update slots=x to work
+ UPDATE Media SET InChanger=0,Slot=0 WHERE InChanger>0 AND Slot>0; (MySQL)
+
+- Add recycle event.
+- Add scratch pool event.
+- Implement NeedVolume event
+- Add Win32 FileSet definition somewhere
+
+
+Document:
+- Does ClientRunAfterJob fail the job on a bad return code?
+- datadir for po files.
+- AM_GNU_GETTEXT finds the library if you specify
+ --with-libintl-prefix
+- Document cleaning up the spool files:
+ db, pid, state, bsr, mail, conmsg, spool
+- Document the multiple-drive-changer.txt script.
+- Pruning with Admin job.
+- Restore of all files for a Job or set of jobs even if the file
+ records have been removed from the catalog.
+- Does WildFile match against full name? Doc.
+
+For 1.39:
+- Add ACL to restore only to original location.
+- Add a recursive mark command (rmark) to restore.
+- "Minimum Job Interval = nnn" sets minimum interval between Jobs
+ of the same level and does not permit multiple simultaneous
+ running of that Job (i.e. lets any previous invocation finish
+ before doing Interval testing).
+- Look at simplifying File exclusions.
+- Fix store_yesno to be store_bitmask.
+- New directive "Delete purged Volumes"
+- new pool XXX with ScratchPoolId = MyScratchPool's PoolId and
+ let it fill itself, and RecyclePoolId = XXX's PoolId so I can
+ see if it become stable and I just have to supervise
+ MyScratchPool
+- If I want to remove this pool, I set RecyclePoolId = MyScratchPool's
+ PoolId, and when it is empty remove it.
+- Figure out how to recycle Scratch volumes back to the Scratch
+ Pool.
+- Add Volume=SCRTCH
+- Allow Check Labels to be used with Bacula labels.
+- "Resuming" a failed backup (lost line for example) by using the
+ failed backup as a sort of "base" job.
+- Look at NDMP
+- Email to the user when the tape is about to need changing x
+ days before it needs changing.
+- Command to show next tape that will be used for a job even
+ if the job is not scheduled.
+--- create_file.c.orig Fri Jul 8 12:13:05 2005
++++ create_file.c Fri Jul 8 12:13:07 2005
+@@ -195,6 +195,8 @@
+ attr->ofname, be.strerror());
+ return CF_ERROR;
+ }
++ } else if(S_ISSOCK(attr->statp.st_mode)) {
++ Dmsg1(200, "Skipping socket: %s\n", attr->ofname);
+ } else {
+ Dmsg1(200, "Restore node: %s\n", attr->ofname);
+ if (mknod(attr->ofname, attr->statp.st_mode, attr->statp.st_rdev) != 0 && errno != EEXIST) {
+- From: Arunav Mandal <amandal@trolltech.com>
+ 1. When jobs are running and bacula for some reason crashes or if I do a
+ restart it remembers and jobs it was running before it crashed or restarted
+ as of now I loose all jobs if I restart it.
+
+ 2. When spooling and in the midway if client is disconnected for instance a
+ laptop bacula completely discard the spool. It will be nice if it can write
+ that spool to tape so there will be some backups for that client if not all.
+
+ 3. We have around 150 clients machines it will be nice to have a option to
+ upgrade all the client machines bacula version automatically.
+
+ 4. Atleast one connection should be reserved for the bconsole so at heavy load
+ I should connect to the director via bconsole which at sometimes I can't
+
+ 5. Another most important feature that is missing, say at 10am I manually
+ started backup of client abc and it was a full backup since client abc has
+ no backup history and at 10.30am bacula again automatically started backup of
+ client abc as that was in the schedule. So now we have 2 multiple Full
+ backups of the same client and if we again try to start a full backup of
+ client backup abc bacula won't complain. That should be fixed.
+
+- Fix bpipe.c so that it does not modify results pointer.
+ ***FIXME*** calling sequence should be changed.
+1.xx Major Projects:
#3 Migration (Move, Copy, Archive Jobs)
- (probably not this version)
#7 Single Job Writing to Multiple Storage Devices
- (probably not this version)
-
-## Create a new GUI chapter explaining all the GUI programs.
-
-Autochangers:
-- Make "update slots" when pointing to Autochanger, remove
- all Volumes from other drives. "update slots all-drives"?
-
-For 1.37:
-- After rename
- 04-Jul 13:01 MainSD: Rufus.2005-07-04_01.05.02 Warning: Director wanted Volume
- "DLT-13Feb04".
- Current Volume "DLT-04Jul05" not acceptable because:
- 1997 Volume "DLT-13Feb04" not in catalog.
- 04-Jul 13:01 MainSD: Please mount Volume "DLT-04Jul05" on Storage Device
- "HP DLT 80" (/dev/nst0) for Job Rufus.2005-07-04_01.05.02
+- Reserve blocks other restore jobs when first cannot connect
+ to SD.
+- Add true/false to conf same as yes/no
+- For Windows disaster recovery see http://unattended.sf.net/
+- regardless of the retention period, Bacula will not prune the
+ last Full, Diff, or Inc File data until a month after the
+ retention period for the last Full backup that was done.
+- update volume=xxx --- add status=Full
- Remove old spool files on startup.
- Exclude SD spool/working directory.
-- Finish TLS implementation.
- Refuse to prune last valid Full backup. Same goes for Catalog.
-- --without-openssl breaks at least on Solaris.
- Python:
- Make a callback when Rerun failed levels is called.
- Give Python program access to Scheduled jobs.
resources were locked.
- The last part is left in the spool dir.
-Document:
-- Port limiting -m in iptables to prevent DoS attacks
- could cause broken pipes on Bacula.
-- Document that Bootstrap files can be written with cataloging
- turned off.
-- Pruning with Admin job.
-- Add better documentation on how restores can be done
-- OS linux 2.4
- 1) ADIC, DLT, FastStor 4000, 7*20GB
- 2) Sun, DDS, (Suns name unknown - Archive Python DDS drive), 1.2GB
- 3) Wangtek, QIC, 6525ES, 525MB (fixed block size 1k, block size etc.
- driver dependent - aic7xxx works, ncr53c8xx with problems)
- 4) HP, DDS-2, C1553A, 6*4GB
-- Doc the following
- to activate, check or disable the hardware compression feature on my
- exb-8900 i use the exabyte "MammothTool" you can get it here:
- http://www.exabyte.com/support/online/downloads/index.cfm
- There is a solaris version of this tool. With option -C 0 or 1 you can
- disable or activate compression. Start this tool without any options for
- a small reference.
-- Linux Sony LIB-D81, AIT-3 library works.
-- Document PostgreSQL performance problems bug 131.
-- Document testing
-- Document that ChangerDevice is used for Alert command.
-- Document new CDROM directory.
-- Document Heartbeat Interval in the dealing with firewalls section.
-- Document the multiple-drive-changer.txt script.
-Maybe in 1.37:
- In restore don't compare byte count on a raw device -- directory
entry does not contain bytes.
- To mark files as deleted, run essentially a Verify to disk, and
- Bug: if a job is manually scheduled to run later, it does not appear
in any status report and cannot be cancelled.
+==== Keeping track of deleted files ====
+ My "trick" for keeping track of deletions is the following.
+ Assuming the user turns on this option, after all the files
+ have been backed up, but before the job has terminated, the
+ FD will make a pass through all the files and send their
+ names to the DIR (*exactly* the same as what a Verify job
+ currently does). This will probably be done at the same
+ time the files are being sent to the SD avoiding a second
+ pass. The DIR will then compare that to what is stored in
+ the catalog. Any files in the catalog but not in what the
+ FD sent will receive a catalog File entry that indicates
+ that at that point in time the file was deleted.
+
+ During a restore, any file initially picked up by some
+ backup (Full, ...) then subsequently having a File entry
+ marked "delete" will be removed from the tree, so will not
+ be restored. If a file with the same name is later OK it
+ will be inserted in the tree -- this already happens. All
+ will be consistent except for possible changes during the
+ running of the FD.
+
+ Since I'm on the subject, some of you may be wondering what
+ the utility of the in memory tree is if you are going to
+ restore everything (at least it comes up from time to time
+ on the list). Well, it is still *very* useful because it
+ allows only the last item found for a particular filename
+ (full path) to be entered into the tree, and thus if a file
+ is backed up 10 times, only the last copy will be restored.
+ I recently (last Friday) restored a complete directory, and
+ the Full and all the Differential and Incremental backups
+ spanned 3 Volumes. The first Volume was not even mounted
+ because all the files had been updated and hence backed up
+ since the Full backup was made. In this case, the tree
+ saved me a *lot* of time.
+
+ Make sure this information is stored on the tape too so
+ that it can be restored directly from the tape.
+
+ Comments from Martin Simmons (I think they are all covered):
+ Ok, that should cover the basics. There are few issues though:
+
+ - Restore will depend on the catalog. I think it is better to include the
+ extra data in the backup as well, so it can be seen by bscan and bextract.
+
+ - I'm not sure if it will preserve multiple hard links to the same inode. Or
+ maybe adding or removing links will cause the data to be dumped again?
+
+ - I'm not sure if it will handle renamed directories. Possibly it will work
+ by dumping the whole tree under a renamed directory?
+
+ - It remains to be seen how the backup performance of the DIR's will be
+ affected when comparing the catalog for a large filesystem.
+
+=====
+
Regression tests:
- Add Pool/Storage override regression test.
- Add delete JobId to regression.
- Review all items in "restore".
- Fix PostgreSQL GROUP BY problems in restore.
- Fix PostgreSQL sql problems in bugs.
+- After rename
+ 04-Jul 13:01 MainSD: Rufus.2005-07-04_01.05.02 Warning: Director wanted Volume
+ "DLT-13Feb04".
+ Current Volume "DLT-04Jul05" not acceptable because:
+ 1997 Volume "DLT-13Feb04" not in catalog.
+ 04-Jul 13:01 MainSD: Please mount Volume "DLT-04Jul05" on Storage Device
+ "HP DLT 80" (/dev/nst0) for Job Rufus.2005-07-04_01.05.02
+## Create a new GUI chapter explaining all the GUI programs.
+- Make "update slots" when pointing to Autochanger, remove
+ all Volumes from other drives. "update slots all-drives"?
+ No, this is done by modifying mtx-changer to list what is
+ in the drives.
+- Finish TLS implementation.
+- Port limiting -m in iptables to prevent DoS attacks
+ could cause broken pipes on Bacula.
+6. Build and test the Volume Shadow Copy (VSS) for Win32.
+- Allow cancel of unknown Job
+- State not saved when closing Win32 FD by icon
+- bsr-opt-test fails. bsr deleted. Fix.
+- Move Python daemon variables from Job to Bacula object.
+ WorkingDir, ConfigFile
+- Document that Bootstrap files can be written with cataloging
+ turned off.
+- Document details of ANSI/IBM labels
+- OS linux 2.4
+ 1) ADIC, DLT, FastStor 4000, 7*20GB
+- Linux Sony LIB-D81, AIT-3 library works.
+- Doc the following
+ to activate, check or disable the hardware compression feature on my
+ exb-8900 i use the exabyte "MammothTool" you can get it here:
+ http://www.exabyte.com/support/online/downloads/index.cfm
+ There is a solaris version of this tool. With option -C 0 or 1 you can
+ disable or activate compression. Start this tool without any options for
+ a small reference.
+- Document Heartbeat Interval in the dealing with firewalls section.
+- Document new CDROM directory.
+- On Win32 working directory must have drive letter ????
+- On Win32 working directory must be writable by SYSTEM to
+ do restores.
+- Document that ChangerDevice is used for Alert command.
+- Add better documentation on how restores can be done
+8. Take one more try at making DVD writing work (no go)
+7. Write a bacula-web document
+- Why isn't the DEVICE structure defined when doing
+ a reservation?
+- Multi-drive changer seems to only use drive 0
+ Multiple drives don't seem to be opened.
+- My database is growing
+- Call GetLastError() in the berrno constructor rather
+ than delaying until strerror.
+- Tape xxx in drive 0, requested in drive 1
+- The mount command does not work with drives other than 0.
+- A mount should cause the SD to re-examine what Slot is
+ loaded.
+- The SD locks on to the first available drive then
+ wants a Volume that is released but in another drive --
+ chaos.
+- Run the regression scripts on Solaris and FreeBSD