Kern's ToDo List
- 22 February 2006
+ 12 November 2006
Major development:
Project Developer
- Pruning with Admin job.
- Does WildFile match against full name? Doc.
- %d and %v only valid on Director, not for ClientRunBefore/After.
+- During tests with the 260 char fix code, I found one problem:
+ if the system "sees" a long path once, it seems to forget it's
+ working drive (e.g. c:\), which will lead to a problem during
+ the next job (create bootstrap file will fail). Here is the
+ workaround: specify absolute working and pid directory in
+ bacula-fd.conf (e.g. c:\bacula\working instead of
+ \bacula\working).
+- Document techniques for restoring large numbers of files.
+- Document setting my.cnf to big file usage.
+- Add example of proper index output to doc. show index from File;
+- Correct the Include syntax in the m4.xxx files in examples/conf
+- Document JobStatus and Termination codes.
+- Fix the error with the "DVI file can't be opened" while
+ building the French PDF.
+- Document more DVD stuff
+- Doc
+ { "JobErrors", "i"},
+ { "JobFiles", "i"},
+ { "SDJobFiles", "i"},
+ { "SDErrors", "i"},
+ { "FDJobStatus","s"},
+ { "SDJobStatus","s"},
+- Document all the little details of setting up certificates for
+ the Bacula data encryption code.
+- Document more precisely how to use master keys -- especially
+ for disaster recovery.
+
Priority:
+- Migration Volume span bug
+- Rescue release
+- Bug reports
+- Test FIFO backup/restore -- make regression
+- Doc items
+- Add encryption regression tests
+- Test Volume compatibility between machine architectures
+- Encryption documentation
+- Wrong jobbytes with query 12 (todo)
+- bacula-1.38.2-ssl.patch
+- Bare-metal recovery Windows (todo)
+- Document need for UTF-8 format
+
+
For 1.39:
-- Fix re-read of last block to check if job has actually written
- a block, and check if block was written by a different job
- (i.e. multiple simultaneous jobs writing).
-- JobStatus and Termination codes.
+- Implement Python event for backing up/restoring a file.
+- Change dbcheck to tell users to use native tools for fixing
+ broken databases, and to ensure they have the proper indexes.
+- add udev rules for Bacula devices.
+- If a job terminates, the DIR connection can close before the
+ Volume info is updated, leaving the File count wrong.
+- Look at why SIGPIPE during connection can cause seg fault in
+ writing the daemon message, when Dir dropped to bacula:bacula
+- Look at zlib 32 => 64 problems.
+- Try turning on disk seek code.
+- Possibly turn on St. Bernard code.
+- Fix bextract to restore ACLs, or better yet, use common routines.
+- Do we migrate appendable Volumes?
+- Remove queue.c code.
- Some users claim that they must do two prune commands to get a
Volume marked as purged.
- Print warning message if LANG environment variable does not specify
UTF-8.
+- New dot commands from Arno.
+ .show device=xxx lists information from one storage device, including
+ devices (I'm not even sure that information exists in the DIR...)
+ .move eject device=xxx mostly the same as 'unmount xxx' but perhaps with
+ better machine-readable output like "Ok" or "Error busy"
+ .move eject device=xxx toslot=yyy the same as above, but with a new
+ target slot. The catalog should be updated accordingly.
+ .move transfer device=xxx fromslot=yyy toslot=zzz
+
+Low priority:
+- Check to see if jcr->stime is lost during rescheduling of
+ jobs in jobq.c
+- Fix re-read of last block to check if job has actually written
+ a block, and check if block was written by a different job
+ (i.e. multiple simultaneous jobs writing).
+- Figure out how to configure query.sql. Suggestion to use m4:
+ == changequote.m4 ===
+ changequote(`[',`]')dnl
+ ==== query.sql.in ===
+ :List next 20 volumes to expire
+ SELECT
+ Pool.Name AS PoolName,
+ Media.VolumeName,
+ Media.VolStatus,
+ Media.MediaType,
+ ifdef([MySQL],
+ [ FROM_UNIXTIME(UNIX_TIMESTAMP(Media.LastWritten) Media.VolRetention) AS Expire, ])dnl
+ ifdef([PostgreSQL],
+ [ media.lastwritten + interval '1 second' * media.volretention as expire, ])dnl
+ Media.LastWritten
+ FROM Pool
+ LEFT JOIN Media
+ ON Media.PoolId=Pool.PoolId
+ WHERE Media.LastWritten>0
+ ORDER BY Expire
+ LIMIT 20;
+ ====
+ Command: m4 -DmySQL changequote.m4 query.sql.in >query.sql
+
+ The problem is that it requires m4, which is not present on all machines
+ at ./configure time.
+- Get Perl replacement for bregex.c
+- Given all the problems with FIFOs, I think the solution is to do something a
+ little different, though I will look at the code and see if there is not some
+ simple solution (i.e. some bug that was introduced). What might be a better
+ solution would be to use a FIFO as a sort of "key" to tell Bacula to read and
+ write data to a program rather than the FIFO. For example, suppose you
+ create a FIFO named:
+
+ /home/kern/my-fifo
+
+ Then, I could imagine if you backup and restore this file with a direct
+ reference as is currently done for fifos, instead, during backup Bacula will
+ execute:
+
+ /home/kern/my-fifo.backup
+
+ and read the data that my-fifo.backup writes to stdout. For restore, Bacula
+ will execute:
+
+ /home/kern/my-fifo.restore
+
+ and send the data backed up to stdout. These programs can either be an
+ executable or a shell script and they need only read/write to stdin/stdout.
+
+ I think this would give a lot of flexibility to the user without making any
+ significant changes to Bacula.
+
+
+==== SQL
+# get null file
+select FilenameId from Filename where Name='';
+# Get list of all directories referenced in a Backup.
+select Path.Path from Path,File where File.JobId=nnn and
+ File.FilenameId=(FilenameId-from-above) and File.PathId=Path.PathId
+ order by Path.Path ASC;
+
+- Look into using Dart for testing
+ http://public.kitware.com/Dart/HTML/Index.shtml
+
+- Look into replacing autotools with cmake
+ http://www.cmake.org/HTML/Index.html
+
=== Migration from David ===
What I'd like to see:
Here's how I'd define Migration Selection Types:
+With Regexes:
+Client -- Migrate data from selected client only. Migration Selection
+Pattern regexp provides pattern to select client names, eg ^FS00* makes
+all client names starting with FS00 eligible for migration.
+
+Jobname -- Migration all jobs matching name. Migration Selection Pattern
+regexp provides pattern to select jobnames existing in pool.
+
+Volume -- Migrate all data on specified volumes. Migration Selection
+Pattern regexp provides selection criteria for volumes to be migrated.
+Volumes must exist in pool to be eligible for migration.
+
+
+With Regex optional:
LowestUtil -- Identify the volume in the pool with the least data on it
and empty it. No Migration Selection Pattern required.
<lowmig>. Pool highmig and lowmig values are in pool definition, no
Migration Selection Pattern required.
-Client -- Migrate data from selected client only. Migration Selection
-Pattern regexp provides pattern to select client names, eg ^FS00* makes
-all client names starting with FS00 eligible for migration.
+
+No regex:
+SQLQuery -- Migrate all jobuids returned by the supplied SQL query.
+Migration Selection Pattern contains SQL query to execute; should return
+a list of 1 or more jobuids to migrate.
PoolResidence -- Migrate data sitting in pool for longer than
PoolResidence value in pool definition. Migration Selection Pattern
optional; if specified, override value in pool definition (value in
minutes).
-Volume -- Migrate all data on specified volumes. Migration Selection
-Pattern regexp provides selection criteria for volumes to be migrated.
-Volumes must exist in pool to be eligible for migration.
-
-Jobname -- Migration all jobs matching name. Migration Selection Pattern
-regexp provides pattern to select jobnames existing in pool.
-
-SQLQuery -- Migrate all jobuids returned by the supplied SQL query.
-Migration Selection Pattern contains SQL query to execute; should return
-a list of 1 or more jobuids to migrate.
[ possibly a Python event -- kes ]
===
+- Mount on an Autochanger with no tape in the drive causes:
+ Automatically selected Storage: LTO-changer
+ Enter autochanger drive[0]: 0
+ 3301 Issuing autochanger "loaded drive 0" command.
+ 3302 Autochanger "loaded drive 0", result: nothing loaded.
+ 3301 Issuing autochanger "loaded drive 0" command.
+ 3302 Autochanger "loaded drive 0", result: nothing loaded.
+ 3902 Cannot mount Volume on Storage Device "LTO-Drive1" (/dev/nst0) because:
+ Couldn't rewind device "LTO-Drive1" (/dev/nst0): ERR=dev.c:678 Rewind error on "LTO-Drive1" (/dev/nst0). ERR=No medium found.
+ 3905 Device "LTO-Drive1" (/dev/nst0) open but no Bacula volume is mounted.
+ If this is not a blank tape, try unmounting and remounting the Volume.
+- If Drive 0 is blocked, and drive 1 is set "Autoselect=no", drive 1 will
+ be used.
+- Autochanger did not change volumes.
+ select * from Storage;
+ +-----------+-------------+-------------+
+ | StorageId | Name | AutoChanger |
+ +-----------+-------------+-------------+
+ | 1 | LTO-changer | 0 |
+ +-----------+-------------+-------------+
+ 05-May 03:50 roxie-sd: 3302 Autochanger "loaded drive 0", result is Slot 11.
+ 05-May 03:50 roxie-sd: Tibs.2006-05-05_03.05.02 Warning: Director wanted Volume "LT
+ Current Volume "LT0-002" not acceptable because:
+ 1997 Volume "LT0-002" not in catalog.
+ 05-May 03:50 roxie-sd: Tibs.2006-05-05_03.05.02 Error: Autochanger Volume "LT0-002"
+ Setting InChanger to zero in catalog.
+ 05-May 03:50 roxie-dir: Tibs.2006-05-05_03.05.02 Error: Unable to get Media record
+
+ 05-May 03:50 roxie-sd: Tibs.2006-05-05_03.05.02 Fatal error: Error getting Volume i
+ 05-May 03:50 roxie-sd: Tibs.2006-05-05_03.05.02 Fatal error: Job 530 canceled.
+ 05-May 03:50 roxie-sd: Tibs.2006-05-05_03.05.02 Fatal error: spool.c:249 Fatal appe
+ 05-May 03:49 Tibs: Tibs.2006-05-05_03.05.02 Fatal error: c:\cygwin\home\kern\bacula
+ , got
+ (missing)
+ llist volume=LTO-002
+ MediaId: 6
+ VolumeName: LTO-002
+ Slot: 0
+ PoolId: 1
+ MediaType: LTO-2
+ FirstWritten: 2006-05-05 03:11:54
+ LastWritten: 2006-05-05 03:50:23
+ LabelDate: 2005-12-26 16:52:40
+ VolJobs: 1
+ VolFiles: 0
+ VolBlocks: 1
+ VolMounts: 0
+ VolBytes: 206
+ VolErrors: 0
+ VolWrites: 0
+ VolCapacityBytes: 0
+ VolStatus:
+ Recycle: 1
+ VolRetention: 31,536,000
+ VolUseDuration: 0
+ MaxVolJobs: 0
+ MaxVolFiles: 0
+ MaxVolBytes: 0
+ InChanger: 0
+ EndFile: 0
+ EndBlock: 0
+ VolParts: 0
+ LabelType: 0
+ StorageId: 1
+
+ Note VolStatus is blank!!!!!
+ llist volume=LTO-003
+ MediaId: 7
+ VolumeName: LTO-003
+ Slot: 12
+ PoolId: 1
+ MediaType: LTO-2
+ FirstWritten: 0000-00-00 00:00:00
+ LastWritten: 0000-00-00 00:00:00
+ LabelDate: 2005-12-26 16:52:40
+ VolJobs: 0
+ VolFiles: 0
+ VolBlocks: 0
+ VolMounts: 0
+ VolBytes: 1
+ VolErrors: 0
+ VolWrites: 0
+ VolCapacityBytes: 0
+ VolStatus: Append
+ Recycle: 1
+ VolRetention: 31,536,000
+ VolUseDuration: 0
+ MaxVolJobs: 0
+ MaxVolFiles: 0
+ MaxVolBytes: 0
+ InChanger: 0
+ EndFile: 0
+ EndBlock: 0
+ VolParts: 0
+ LabelType: 0
+ StorageId: 1
+===
+ mount
+ Automatically selected Storage: LTO-changer
+ Enter autochanger drive[0]: 0
+ 3301 Issuing autochanger "loaded drive 0" command.
+ 3302 Autochanger "loaded drive 0", result: nothing loaded.
+ 3301 Issuing autochanger "loaded drive 0" command.
+ 3302 Autochanger "loaded drive 0", result: nothing loaded.
+ 3902 Cannot mount Volume on Storage Device "LTO-Drive1" (/dev/nst0) because:
+ Couldn't rewind device "LTO-Drive1" (/dev/nst0): ERR=dev.c:678 Rewind error on "LTO-Drive1" (/dev/nst0). ERR=No medium found.
+
+ 3905 Device "LTO-Drive1" (/dev/nst0) open but no Bacula volume is mounted.
+ If this is not a blank tape, try unmounting and remounting the Volume.
+
+- Add VolumeState (enable, disable, archive)
+- Add VolumeLock to prevent all but lock holder (SD) from updating
+ the Volume data (with the exception of VolumeState).
+- The btape fill command does not seem to use the Autochanger
+- Make Windows installer default to system disk drive.
+- Look at using ioctl(FIOBMAP, ...) on Linux, and
+ DeviceIoControl(..., FSCTL_QUERY_ALLOCATED_RANGES, ...) on
+ Win32 for sparse files.
+ http://www.flexhex.com/docs/articles/sparse-files.phtml
+ http://www.informatik.uni-frankfurt.de/~loizides/reiserfs/fibmap.html
+- Directive: at <event> "command"
+- Command: pycmd "command" generates "command" event. How to
+ attach to a specific job?
+- Integrate Christopher's St. Bernard code.
+- run_cmd() returns int should return JobId_t
+- get_next_jobid_from_list() returns int should return JobId_t
- Document export LDFLAGS=-L/usr/lib64
- Don't attempt to restore from "Disabled" Volumes.
- Network error on Win32 should set Win32 error code.
then be migrated.
- Detect resource deadlock in Migrate when same job wants to read
and write the same device.
-- Make hardlink code at line 240 of find_one.c use binary search.
- Queue warning/error messages during restore so that they
are reported at the end of the report rather than being
hidden in the file listing ...
- Look at fixing restore status stats in SD.
- Make selection of Database used in restore correspond to
client.
+- Look at using ioctl(FIMAP) and FIGETBSZ for sparse files.
+ http://www.informatik.uni-frankfurt.de/~loizides/reiserfs/fibmap.html
- Implement a mode that says when a hard read error is
encountered, read many times (as it currently does), and if the
block cannot be read, skip to the next block, and try again. If
("D","Diff"),
("I","Inc");
- Add ACL to restore only to original location.
+- Show files/second in client status output.
- Add a recursive mark command (rmark) to restore.
- "Minimum Job Interval = nnn" sets minimum interval between Jobs
of the same level and does not permit multiple simultaneous
- In restore don't compare byte count on a raw device -- directory
entry does not contain bytes.
-- To mark files as deleted, run essentially a Verify to disk, and
- when a file is found missing (MarkId != JobId), then create
- a new File record with FileIndex == -1. This could be done
- by the FD at the same time as the backup.
=== rate design
jcr->last_rate
jcr->last_runtime
- Bug: if a job is manually scheduled to run later, it does not appear
in any status report and cannot be cancelled.
-==== Keeping track of deleted files ====
+==== Keeping track of deleted/new files ====
+- To mark files as deleted, run essentially a Verify to disk, and
+ when a file is found missing (MarkId != JobId), then create
+ a new File record with FileIndex == -1. This could be done
+ by the FD at the same time as the backup.
+
My "trick" for keeping track of deletions is the following.
Assuming the user turns on this option, after all the files
have been backed up, but before the job has terminated, the
pass. The DIR will then compare that to what is stored in
the catalog. Any files in the catalog but not in what the
FD sent will receive a catalog File entry that indicates
- that at that point in time the file was deleted.
+ that at that point in time the file was deleted. This
+ either transmitted to the FD or simultaneously computed in
+ the FD, so that the FD can put a record on the tape that
+ indicates that the file has been deleted at this point.
+ A delete file entry could potentially be one with a FileIndex
+ of 0 or perhaps -1 (need to check if FileIndex is used for
+ some other thing as many of the Bacula fields are "overloaded"
+ in the SD).
During a restore, any file initially picked up by some
backup (Full, ...) then subsequently having a File entry
Make sure this information is stored on the tape too so
that it can be restored directly from the tape.
+ All the code (with the exception of formally generating and
+ saving the delete file entries) already exists in the Verify
+ Catalog command. It explicitly recognizes added/deleted files since
+ the last InitCatalog. It is more or less a "simple" matter of
+ taking that code and adapting it slightly to work for backups.
+
Comments from Martin Simmons (I think they are all covered):
Ok, that should cover the basics. There are few issues though:
- Fix Maximum Changer Wait, Maximum Open Wait, Maximum Rewind Wait to
accept time qualifiers.
- Does ClientRunAfterJob fail the job on a bad return code?
+- Make hardlink code at line 240 of find_one.c use binary search.
+- Add ACL error messages in src/filed/acl.c.
+- Make authentication failures single threaded.
+- Make Dir and SD authentication errors single threaded.
+- Fix catreq.c digestbuf at line 411 in src/dird/catreq.c
+- Make base64.c (bin_to_base64) take a buffer length
+ argument to avoid overruns.
+ and verify that other buffers cannot overrun.
+- Implement VolumeState as discussed with Arno.
+- Add LocationId to update volume
+- Add LocationLog
+ LogId
+ Date
+ User text
+ MediaId
+ LocationId
+ NewState???
+- Add Comment to Media record
+- Fix auth compatibility with 1.38
+- Update dbcheck to include Log table
+- Update llist to include new fields.
+- Make unmount unload autochanger. Make mount load slot.
+- Fix bscan to report the JobType when restoring a job.
+- Fix wx-console scanning problem with commas in names.
+- Add manpages to the list of directories for make install. Notify
+ Scott
+- Add bconsole option to use stdin/out instead of conio.
+- Fix ClientRunBefore/AfterJob compatibility.
+- Ensure that connection to daemon failure always indicates what
+ daemon it was trying to connect to.
+- Freespace on DVD requested over and over even with no intervening
+ writes.
+- .update volume [enabled|disabled|*see below]
+ > However, I could easily imagine an option to "update slots" that says
+ > "enable=yes|no" that would automatically enable or disable all the Volumes
+ > found in the autochanger. This will permit the user to optionally mark all
+ > the Volumes in the magazine disabled prior to taking them offsite, and mark
+ > them all enabled when bringing them back on site. Coupled with the options
+ > to the slots keyword, you can apply the enable/disable to any or all volumes.
+- Restricted consoles start in the Default catalog even if it
+ is not permitted.
+- When reading through parts on the DVD, the DVD is mounted and
+ unmounted for each part.
+- Make sure that the restore options don't permit "seeing" other
+ Client's job data.
+- Restore of a raw drive should not try to check the volume size.
+- Lock tape drive door when open()
+- Make release unload any autochanger.
+- Arno's reservation deadlock.
+- Eric's SD patch
projects / bacula / bacula / blobdiff