Kern's ToDo List
- 01 January 2003
+ 25 January 2003
Documentation to do: (a little bit at a time)
- Document running a test version.
-- Make sure restore options are documented
- Document query file format.
- Document static linking
-- Document nofollow flag
+- Document how to automatically backup all local partitions
+- Document problems with Verify and pruning.
+- Document how to use multiple databases.
+
Testing to do: (painful)
- that console command line options work
- blocksize recognition code.
-For 1.28 release:
-- Look at ua_prune.c in detail. Why did JobType work at all??????
-- Figure out how to allow multiple simultaneous file Volumes on
- a single device.
-- Why are save/restore of device different sizes (sparse?)
-- Backup of raw partitions
-
-For 1.29 release:
-- Implement SHA1
+For 1.30 release:
+- Add Signature type to File DB record.
+- CD into subdirectory when open()ing files for backup to
+ speed up things. Test with testfind().
+- Add prefixlinks to where or not where absolute links to FD.
+- Look at handling <> in smtp doesn't work with exim.
+- Priority job to go to top of list.
+- Implement Bar code handling
+- Why is catreq.c:111 Find vol called twice for a job?
+- Find out why Full saves run slower and slower (hashing?)
+- Figure out how to allow multiple simultaneous file Volumes on a single device.
+- Why are save/restore of device different sizes (sparse?) Yup! Fix it.
+- Implement some way for the Console to dynamically create a job.
+- Restore to a particular time -- e.g. before date, after date.
- Implement disk spooling
- Implement finer multiprocessing options.
- Solaris -I on tar for include list
-- Add whereln to where or not where absolute links
-- Figure out some way to automatically backup all local partitions
- Enable avoid backing up archive device (findlib/find_one.c:128)
- Implement FileOptions (see end of this document)
- Implement Bacula plugins -- design API
-- Make hash table for linked files in findlib/find_one.c:161
- Make bcopy read through bad tape records.
- Need a verbose mode in restore, perhaps to bsr.
-- Should we dump a SOS when starting a new tape?
- bscan without -v is too quiet -- perhaps show jobs.
- Add code to reject whole blocks if not wanted on restore.
- Implement multiple simultaneous file Volumes on a single device.
- Restore program that errors in SD due to no tape reports
OK incorrectly in output.
- After unmount, if restore job started, ask to mount.
-- Fix db_get_fileset in cats/sql_get.c for multiple records.
-- Fix catalog filename truncation in sql_get and sql_create. Use
- only a single filename split routine.
- Make Restore report an error if FD or SD term codes are not OK.
- Convert all %x substitution variables, which are hard to remember
and read to %(variable-name). Idea from TMDA.
- Add JobLevel in FD status (but make sure it is defined).
- Make Pool resource handle Counter resources.
- Remove NextId for SQLite. Optimize.
-- Fix gethostbyname() to use gethostbyname_r()
- Strip trailing / from Include
- Move all SQL statements into a single location.
- Cleanup db_update_media and db_update_pool
- Add Client FS/OS id (Linux, Win95/98, ...).
- Test a second language e.g. french.
- Compare tape to Client files (attributes, or attributes and data)
-- Restore to a particular time -- e.g. before date, after date.
- Make all database Ids 64 bit.
- Write an applet for Linux.
- Add estimate to Console commands
- Fix Win2000 error with no messages during startup.
- Events : tape has more than xxx bytes.
- Restrict characters permitted in a Resource name.
-- Complete code in Bacula Resources -- this will permit
+- Complete code in Bacula Resources -- this will permit
reading a new config file at any time.
- Handle ctl-c in Console
- Implement LabelTemplate (at least first cut).
- Implement script driven addition of File daemon to config files.
- Think about how to make Bacula work better with File (non-tape) archives.
+- Write Unix emulator for Windows.
-- see setgroup and user for Bacula p4-5 of stunnel.c
- Implement new serialize subroutines
send(socket, "string", &Vol, "uint32", &i, NULL)
- Audit all UA commands to ensure that we always prompt where possible.
- Make database type selectable by .conf files i.e. at runtime
- gethostbyname failure in bnet_connect() continues
generating errors -- should stop.
-- Add HOST to Volume label.
- Set flag for uname -a. Add to Volume label.
- Implement throttled work queue.
- Check for EOT at ENOSPC or EIO or ENXIO (unix Pc)
- Make bcopy copy with a single tape drive.
- Permit changing ownership during restore.
-- Restore should get Device and Pool information from
- job record rather than from config.
- Autolabel should be specified by DIR instead of SD.
- Find out how to get the system tape block limits, e.g.:
Apr 22 21:22:10 polymatou kernel: st1: Block limits 1 - 245760 bytes.
=====
- FD sends unsaved file list to Director at end of job (see
RFC below).
+- File daemon should build list of files skipped, and then
+ at end of save retry and report any errors.
- Write a Storage daemon that uses pipes and
standard Unix programs to write to the tape.
See afbackup.
if tape is not valid.
- Verify from Volume
- Ensure that /dev/null works
-- File daemon should build list of files skipped, and then
- at end of save retry and report any errors.
- Need report class for messages. Perhaps
report resource where report=group of messages
- enhance scan_attrib and rename scan_jobtype, and
fill in code for "since" option
-- Need to save contents of FileSet to tape?
- Director needs a time after which the report status is sent
anyway -- or better yet, a retry time for the job.
Don't reschedule a job if previous incarnation is still running.
-- Figure out how to save the catalog (possibly a special FileSet).
-- Figure out how to restore the catalog.
- Some way to automatically backup everything is needed????
- Need a structure for pending actions:
- buffered messages
is NOT currently the case). Must detect socket error,
buffer messages for later.
- Enhance time/duration input to allow multiple qualifiers e.g. 3d2h
+- Add ability to backup to two Storage devices (two SD sessions) at
+ the same time -- e.g. onsite, offsite.
+- Add the ability to consolidate old backup sets (basically do a restore
+ to tape and appropriately update the catalog). Compress Volume sets.
+ Might need to spool via file is only one drive is available.
+- Compress or consolidate Volumes of old possibly deleted files. Perhaps
+ someway to do so with every volume that has less than x% valid
+ files.
Projects:
Bacula Projects Roadmap
17 August 2002
- last update 27 November 2002
+ last update 5 January 2003
Item 1: Multiple simultaneous Jobs. (done)
-Done
+Done -- Restore part needs better implementation to work correctly
What: Permit multiple simultaneous jobs in Bacula.
applied for the backup.
====================================
+=========================================
+Proposal by Bill Sellers
+
+Return-Path: <w.a.sellers@larc.nasa.gov>
+Received: from post.larc.nasa.gov (post.larc.nasa.gov [128.155.4.45]) by matou.sibbald.com (8.11.6/8.11.6) with ESMTP id h0ELUIm07622 for <kern@sibbald.com>; Tue, 14 Jan 2003 22:30:18 +0100
+Received: from Baron.larc.nasa.gov (baron.larc.nasa.gov [128.155.40.132]) by post.larc.nasa.gov (pohub4.6) with ESMTP id QAA09768 for <kern@sibbald.com>; Tue, 14 Jan 2003 16:30:14 -0500 (EST)
+Message-Id: <5.1.0.14.2.20030114153452.028dbae8@pop.larc.nasa.gov>
+X-Sender: w.a.sellers@pop.larc.nasa.gov
+X-Mailer: QUALCOMM Windows Eudora Version 5.1
+Date: Tue, 14 Jan 2003 16:30:18 -0500
+To: Kern Sibbald <kern@sibbald.com>
+From: Bill Sellers <w.a.sellers@larc.nasa.gov>
+Subject: Re: [Bacula-users] Bacula remote storage?
+In-Reply-To: <1042565382.1845.177.camel@rufus>
+References: <5.1.0.14.2.20030114113004.0293a210@pop.larc.nasa.gov> <5.1.0.14.2.20030113170650.028dad88@pop.larc.nasa.gov> <5.1.0.14.2.20030113170650.028dad88@pop.larc.nasa.gov> <5.1.0.14.2.20030114113004.0293a210@pop.larc.nasa.gov>
+Mime-Version: 1.0
+Content-Type: text/plain; charset="us-ascii"; format=flowed
+X-Annoyance-Filter-Junk-Probability: 0
+X-Annoyance-Filter-Classification: Mail
+At 06:29 PM 1/14/2003 +0100, you wrote:
+>Hello Bill,
+>
+>Well, if you cannot put a Bacula client on the machine,
+>then it is a big problem. If you know of some software
+>that can do what you want, let me know, because I
+>really just don't know how to do it -- at least not
+>directly.
+
+
+Hi Kern,
+
+We have been able to get Amanda to use the HSM as a storage
+device. Someone here wrote a driver for Amanda. BUT, Amanda doesn't
+handle Windows systems very well (or at all without Samba). So I am
+looking for a backup system that has a Windows client. I really like the
+Windows integration of Bacula.
+
+From the command line, its rather trivial to move the data around. We use
+something like-
+
+tar cf - ./files | gzip -c | rsh hsm dd of=path/file.tgz
+
+or if you use GNU tar:
+
+tar czf hsm:path/file.tgz ./files
+
+One idea for you to consider; Sendmail offers pipes in the aliases file;
+(mailpipe: "|/usr/bin/vacation root") and Perl supports pipes in the
+"open" statement (open FILE, "|/bin/nroff -man";) Could you could make a
+pipe available, as a storage device? Then we could use any command that
+handles stdin as a storage destination.
+
+Something like-
+
+Storage {
+ Name = HSM-RSH
+ Address = hsm
+ #Password is not used in rsh, but might be used in ftp.
+ Device = "| gzip -c | rsh hsm dd of=path/file.tgz"
+ MediaType = Pipe
+}
+
+Storage {
+ Name = HSM-FTP
+ Address = hsm
+ Password = "foobar&-"
+ Device = "| ncftpput -c hsm /path/file.bacula"
+ MediaType = Pipe
+}
+
+>If you have some local storage available, you could
+>use Bacula to backup to disk volumes, then use some
+>other software (ftp, scp) to move them to the HSM
+>machine. However, this is a bit kludgy.
+
+
+It is, but maybe worth a try. Is there some function in Bacula to put
+variables in filenames? i.e. backup.2003-01-15.root
+
+Thanks!
+Bill
+
+---
+Bill Sellers
+w.a.sellers@larc.nasa.gov
+
+==============================================
+ The Project for the above
+
+I finally realized that this is not at all
+the same as reader/writer programs or plugins,
+which are alternate ways of accessing the
+files to be backed up. Rather, it is an alternate
+form of storage device, and I have always planned
+that Bacula should be able to handle all sorts
+of storage devices.
+
+So, I propose the following phases:
+
+1. OK from you to invest some time in testing
+ this as I implement it (requires that you
+ know how to download from the SourceForge
+ cvs -- which I imagine is a piece of cake
+ for you).
+
+2. Dumb implementation by allowing a device to
+ be a fifo for write only.
+ Reason: easy to implement, proof of concept.
+
+3. Try reading from fifo but with fixed block
+ sizes.
+ Reason: proof of concept, easy to implement.
+
+4. Extend reading from fifo (restores) to handle
+ variable blocks.
+ Reason: requires some delicate low level coding
+ which could destabilize all of Bacula.
+
+5. Implementation of above but to a program. E.g.
+ Device = "|program" (not full pipeline).
+ Reason: routines already exist, and program can
+ be a shell script which contains anything.
+
+6. Add full pipeline as a possibility. E.g.
+ Device = "| gzip -c | rsh hsm dd of=path/file.tgz"
+ Reason: needs additional coding to implement full
+ pipeline (must fire off either a shell or all
+ programs and connect their pipes).
+
+There are a good number of details in each step
+that I have left out, but I will specify them at
+every stage, and there may be a few changes as things
+evolve. I expect that to get to stage 5 will take a
+few weeks, and at that point, you will have
+everything you need (just inside a script).
+Stage 6 will probably take longer, but if this
+project pleases you, what we do for 5 should
+be adequate for some time.
+
+
+
+=============================================
+
Done: (see kernsdone for more)
-- Add EOM records? No, not at this time. The current system works and
- above all is simple.
-- Add VolumeUseDuration and MaximumVolumeJobs to Pool db record and
- to Media db record.
-- Add VOLUME_CAT_INFO to the EOS tape record (as
- well as to the EOD record). -- No, not at this time.
-- Put MaximumVolumeSize in Director (MaximumVolumeJobs, MaximumVolumeFiles,
- MaximumFileSize).
-- Enhance schedule to have 1stSat, ...
-- Make sure catalog doesn't keep growing.
-- On I/O error, write EOF, then try to write again ? No, keep it simple.
-- Figure out how compress everything except .gz,... files.
- Implement FileOptions.
-- Put Bacula version somewhere in Job stream, probably Start Session Labels.
-- Fix start/end blocks for File devices
-- Make Job err if WriteBootstrap fails.
-- Test that mod of restore options works.
-- Test that week position schedule code works.
-- Make BSR accept count (total files to be restored).
-- Add code to fast seek to proper place on tape/file when doing Restore.
-- Replace popen() and pclose() -- fail safe and timeout, no SIG dep.
-- Add code to put VolFile in bsr for restore command.
-- Volumes can be listed multiple times in Restore volume list.
-- Add watchdog timeout for child processes start_child_timer()
- end_child_timer();
-- Get rid of bscan.c:534 error message (one time only).
-- Print some statistics when get EOF on device in bscan -- feedback
- to let user know it is working.
-- DateWritten field on tape may be wrong.
-- Ensure that restore of differential jobs works (check SQL).
-- Count number of ignored messages in bscan and print when first SOS is found.
-- Test that EndFile/Block are correctly updated at end of tape
- (in view of new block reading code).
-- Test watchdog child timer code.
-- Test new BSR code (mostly done).
-- Work more on how to to a Bacula restore beginning with
- just a Bacula tape and a boot floppy (bare metal recovery).
-- Restore options (overwrite, overwrite if older,
- overwrite if newer, never overwrite, ...)
-- Fill all fields in Vol/Job Header -- ensure that everything
- needed is written to tape. Think about restore to Catalog
- from tape. Client record needs improving.
-- Implement ./configure --with-client-only
-- Finish up static linking
-- that restore options work in FD
-
+- Look into Pruning/purging problems or why there seem to
+ be so many files listed each night.
+- Fix cancel in find_one -- need jcr.
+- Cancel does not work for restore in FD.
+- Write SetJobStatus() function so cancel status not lost.
+- Add include list to end of chain in findlib
+- Zap sd_auth_key after use
+- Add Bar code reading capabilities (new mtx-changer)
+- Figure out some way to automatically backup all local partitions
+- Make hash table for linked files in findlib/find_one.c:161
+ (not necessary)
+- Rewrite find_one.c to use only pool_memory instead of
+ alloca and malloc (probably not necessary).
+- Make sure btraceback goes into /sbin not sysconf directory.
+- InitVerify is getting pruned and it shouldn't (document it)
+- Make 1.28c release ??? NO do 1.29 directly
+- Set timeout on opening fifo for save or restore (findlib)
+- Document FIFO storage device.
+- Document fifo and | and <
+====== 1.30 =======
+- Implement SHA1
+- Get correct error status from run_program or open_bpipe().
+