X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fkernstodo;h=fadf6e22a617baf7472577fbcb280fafac2c5a44;hb=2d3535a82513c47d34f3016b87099af1bcff8bc2;hp=34db384b9153fd257715d105936a28636b87f4d2;hpb=dffda24066de22996f1a8b1a7958975dcaf2b35c;p=bacula%2Fbacula diff --git a/bacula/kernstodo b/bacula/kernstodo index 34db384b91..fadf6e22a6 100644 --- a/bacula/kernstodo +++ b/bacula/kernstodo @@ -1,8 +1,43 @@ - Kern's ToDo List - 31 March 2004 + Kern's ToDo List + 17 Septermber 2004 + +Major development: +Project Developer +======= ========= +IPv6_2 Meno Abels +Data encryption Meno Abels (see projects) +Communication encryption Meno Abels +Version 1.35 Kern (see below) +======================================================== + +1.35 Items to do for release: +- Backspace to beginning of line (conio) does not erase first char. +- Doc to do unmount before removing magazine. +- Detect fixed tape block mode during positioning by looking at + block numbers in btape "test". Possibly adjust in Bacula. + +- Document a get out of jail procedure if everything breaks if + you lost/broke the Catalog -- do the same for "I know my + file is there how do I get it back?". + +- Add "Rerun failed levels = yes/no" to Job resource. + + +Maybe for 1.35: +- Look at patches/bacula_db.b2z postgresql that loops during restore. + See Gregory Wright. +- Add delete JobId to regression. +- Add bscan to four-concurrent-jobs regression. +- Add IPv6 to regression +- Perhaps add read/write programs and/or plugins to FileSets. +- How to handle backing up portables ... Documentation to do: (any release a little bit at a time) +- Alternative to static linking "ldd prog" save all binaries listed, + restore them and point LD_LIBRARY_PATH to them. +- Document add "/dev/null 2>&1" to the bacula-fd command line - Document query file format. +- Add more documentation for bsr files. - Document problems with Verify and pruning. - Document how to use multiple databases. - VXA drives have a "cleaning required" @@ -22,35 +57,170 @@ Documentation to do: (any release a little bit at a time) changing tape settings while a job is running.) - Lookup HP cleaning recommendations. - Lookup HP tape replacement recommendations (see trouble shooting autochanger) +- Add more obvious documentation of @ for conf files. +- Document doing table repair + Testing to do: (painful) -- Test drive polling! -- blocksize recognition code. -- Test if rewind at end of tape waits for tape to rewind. -- Test cancel at EOM. -For 1.33 Testing/Documentation: -- Figure out how to use ssh or stunnel to protect Bacula communications. - Add Dan's work to manual See ssl.txt + +For 1.37 Testing/Documentation: +- If opening a tape in read/write mode fails attempt to open + it in read-only mode, and mark the tape for read only. +- Add a read-only mode to the mount option. +- Add "Allow multiple connections" in Catalog resource to open a new + database connection for each job. +- Allow Simultaneous Priorities = yes => run up to Max concurrent jobs even + with multiple priorities. +- Fix find_device in stored/dircmd.c:462 (see code) - Add db check test to regression. Test each function like delete, purge, ... - If you use restore replace=never, the directory attributes for non-existent directories will not be restored properly. -1.34 Cleanup +Fix: + Kern, + + mt status and tapeinfo output are attached. Default blocksize was + originally 512 bytes. I ran the following test on my machine: + + - set defblksize to 512 + - started bacula, labelled tape, ran job + - job completes with the following in log + SD termination status: OK + Termination: Backup OK + - restore attempts with blocksize set to 0 or 512 result in the familiar + "Volume data error! Wanted block-id BB02, got . Buffer discarded." + + + # tapeinfo -f /dev/sg0 + Product Type: Tape Drive + Vendor ID: 'SONY ' + Product ID: 'TSL-11000 ' + Revision: 'L100' + Attached Changer: Yes + SerialNumber: '0010000443' + MinBlock:1 + MaxBlock:16777215 + SCSI ID: 1 + SCSI LUN: 0 + Ready: yes + BufferedMode: yes + Medium Type: Not Loaded + Density Code: 0x26 + BlockSize: 512 + DataCompEnabled: no + DataCompCapable: yes + DataDeCompEnabled: yes + CompType: 0x20 + DeCompType: 0x20 + BOP: yes + Block Position: 0 + + # mt -f /dev/nst0 status + SCSI 2 tape drive: + File number=0, block number=132, partition=0. + Tape block size 512 bytes. Density code 0x26 (DDS-4 or QIC-4GB). + Soft error count since last status=0 + General status bits on (1010000): + ONLINE IM_REP_EN + + + +Wish list: +- see lzma401.zip in others directory for new compression + algorithm/library. +- Minimal autochanger handling in Bacula and in btape. +- Look into how tar does not save sockets and the possiblity of + not saving them in Bacula (Martin Simmons reported this). +- Add All Local Partitions = yes to new style saves. +- localmounts=`awk '/ext/ { print $2 }' /proc/mounts` # or whatever + find $localmounts -xdev -type s -ls +- Fix restore jobs so that multiple jobs can run if they + are not using the same tape(s). +- Allow the user to select JobType for manual pruning/purging. +- Look at adding Client run command that will use the + port opened by the client. +- bscan does not put first of two volumes back with all info in + bscan-test. +- Implement the FreeBSD nodump flag in chflags. +- Figure out how to make named console messages go only to that + console and to the non-restricted console (new console class?). +- Make restricted console prompt for password if *ask* is set or + perhaps if password is undefined. +- Implement "from ISO-date/time every x hours/days/weeks/months" in + schedules. +==== from Marc Schoechlin +- the help-command should be more verbose + (it should explain the paramters of the different + commands in detail) + -> it´s time-comsuming to consult the manual anytime + you need a special parameter + -> maybe it´s more easy to maintain this, if the + descriptions of that commands are outsourced to + a ceratin-file +- the cd-command should allow complete paths + i.e. cd /foo/bar/foo/bar + -> if a customer mails me the path to a certain file, + it´s faster to enter the specified directory +- if the password is not configured in bconsole.conf + you should be asked for it. + -> sometimes you like to do restore on a customer-machine + which shouldn´t know the password for bacula. + -> adding the password to the file favours admins + to forget to remove the password after usage + -> security-aspects + the protection of that file is less important +- long-listed-output of commands should be scrollable + like the unix more/less-command does + -> if someone runs 200 and more machines, the lists could + be a little long and complex +- command-output should be shown column by column + to reduce scrolling and to increase clarity + -> see last item +- lsmark should list the selected files with full + paths +- wildcards for selecting and file and directories would be nice +- any actions should be interuptable with STRG+C +- command-expansion would be pretty cool +==== +- Compaction of Disk space by "migrating" Volumes that have pruned + Jobs (what criteria? size, #jobs, time). +- Add prune all command +- Document fact that purge can destroy a part of a restore by purging + one volume while others remain valid -- perhaps mark Jobs. - Add multiple-media-types.txt - look at mxt-changer.html - - -For version 1.35: +- Make ? do a help command (no return needed). +- Implement restore directory. +- Document streams and how to implement them. +- Possibly implement "Ensure Full Backup = yes" looks for a failed full backup + and upgrades the current backup if one exists. +- Check that barcode reading and update slots scan works. +- Try not to re-backup a file if a new hard link is added. +- Add feature to backup hard links only, but not the data. +- Fix stream handling to be simpler. +- Add Priority and Bootstrap to Run a Job. +- Eliminate Restore "Run Restore Job" prompt by allowing new "run command + to be issued" +- Remove View FileSet button from Run a Job dialog. +- Don't count higher level non-saved directories in Files expected + in restore report. +- Handle prompt for restore job at end of Restore command. +- Add display of total selected files to Restore window. +- Add tree pane to left of window. +- Add progress meter. +- Max wait time or max run time causes seg fault -- see runtime-bug.txt +- Document writing to a CD/DVD with Bacula. +- Add a "base" package to the window installer for pthreadsVCE.dll + which is needed by all packages. +- Add message to user to check for fixed block size when the forward + space test fails in btape. - When unmarking a directory check if all files below are unmarked and then remove the + flag -- in the restore tree. - Possibly implement: Action = Unmount Device="TapeDrive1" in Admin jobs. - Setup lrrd graphs: (http://www.linpro.no/projects/lrrd/) Mike Acar. - Revisit the question of multiple Volumes (disk) on a single device. -- Finish SIGHUP work. -- Check that all change in wait status in the SD are - signaled to the Director. - Add a block copy option to bcopy. - Investigate adding Mac Resource Forks. - Finish work on Gnome restore GUI. @@ -61,30 +231,20 @@ For version 1.35: Also 12:00am works fine. 0:00pm also works (which I don't think should). None of the values 12:00pm - 12:59pm work for that matter. - Add level to estimate command. +- For each job type (Admin, Restore, ...) require only the really necessary + fields. - Fix option 2 of restore -- list where file is backed up -- require Client, then list last 20 backups. - Add all pools in Dir conf to DB also update them to catch changed LabelFormats and such. -- Update volumes FromPool=xxx does all volumes. - Pass Director resource name as an option to the Console. - Add a "batch" mode to the Console (no unsolicited queries, ...). -- Add code to check for tape alerts -- tapeinfo. -- Make sure list of Volumes needed is in correct order for restore. - See havana. -- Remove paths (and files that reference them) that have no trailing slash - in dbcheck -- or add a trailing slash. -- Remove Filenames (and files that reference them) that have a trailing - slash in dbcheck -- or remove the trailing slash. -- Remove orphaned paths/filenames by copying them to a new table with a - reference count, then mark all referenced files/paths and remove unreferenced - ones. - Add a .list all files in the restore tree (probably also a list all files) Do both a long and short form. - Allow browsing the catalog to see all versions of a file (with stat data on each file). - Restore attributes of directory if replace=never set but directory did not exist. -- Allow "delete job jobid=xxx,yyy,aaa-bbb" i.e. list + ranges. - Use SHA1 on authentication if possible. - See comtest-xxx.zip for Windows code to talk to USB. - Make btape accept Device Names in addition to Archive names. @@ -105,40 +265,23 @@ For version 1.35: - Add disk seeking on restore. - Allow for optional cancelling of SD and FD in case DIR gets a fatal error. Requested by Jesse Guardiani -- Bizarre message: Error: Could not open WriteBootstrap file: -- Build console in client only build. - Add "limit=n" for "list jobs" - Check new HAVE_WIN32 open bits. - Check if the tape has moved before writing. - Handling removable disks -- see below: -- Multiple drive autochanger support -- see below. - Keep track of tape use time, and report when cleaning is necessary. -- Fix FreeBSD mt_count problem. - Add FromClient and ToClient keywords on restore command (or BackupClient RestoreClient). -- Automatic "update slots" on user configuration directive when a - slot error occurs. - Implement a JobSet, which groups any number of jobs. If the JobSet is started, all the jobs are started together. Allow Pool, Level, and Schedule overrides. - Enhance cancel to timeout BSOCK packets after a specific delay. -- When I restore to Windows the Created, Accessed and Modifiedtimes are - those of the time of the restore, not those of the originalfile. - The dates you will find in your restore log seem to be the original - creation dates -- Volume "add"ed to Pool gets recycled in first use. VolBytes=0 -- If a tape is recycled while it is mounted, Stanislav Tvrudy must do an - additional mount to deblock the job. -- From Johan Decock: - bscan: sql_update.c:65 UPDATE File SET MD5='Ij+5kwN6TFIxK+8l8+/I+A' WHERE FileId=0 - bscan: bscan.c:1074 Could not add MD5/SHA1 to File record. ERR=sql_update.c:65 Update problem: affected_rows=0 - Do scheduling by UTC using gmtime_r() in run_conf, scheduler, and ua_status.!!! Thanks to Alan Brown for this tip. - Look at updating Volume Jobs so that Max Volume Jobs = 1 will work correctly for multiple simultaneous jobs. - Correct code so that FileSet MD5 is calculated for < and | filename generation. -- Mark Volume in error on error from WEOF. - Implement the Media record flag that indicates that the Volume does disk addressing. - Implement VolAddr, which is used when Volume is addressed like a disk, @@ -149,8 +292,6 @@ For version 1.35: - Fix fast block rejection (stored/read_record.c:118). It passes a null pointer (rec) to try_repositioning(). - Look at extracting Win data from BackupRead. -- Having dashes in filenames apparently creates problems for restore - by filename??? hard to believe. - Implement RestoreJobRetention? Maybe better "JobRetention" in a Job, which would take precidence over the Catalog "JobRetention". - Implement Label Format in Add and Label console commands. @@ -162,31 +303,6 @@ For version 1.35: resources, like Level? If so, I think I'd make it an optional directive in Job, Client, and Pool, with precedence such that Job overrides Client which in turn overrides Pool. -- Print a message when a job starts if the conf file is not current. -- To pass Include 1 or two letter commands - I Name Include name - first record - B Name Base name - repeat - R "xxx" Regexp - W "xxx" Wild Card - E zzz Exclude expression (wild card) - P "plugin" Plugin - D "reader" Reader program - T "writer" Writer program - O Options In current commpressed format - (compression, signature, onefs, recurse, sparse, - replace, verify options, ...) - N End option set - B BaseName Start second option set any letter - ... - E - F Number Number of filenames to follow - - B Name - ... - N End option set - F Number Number of filenames to follow - - ... - Spooling ideas taken from Volker Sauer's and other's emails: > IMHO job spooling should be turned on @@ -249,6 +365,11 @@ For version 1.35: from one backup Volume to another. - New Storage specifications: + - Want to write to multiple storage devices simultaneously + - Want to write to multiple storage devices sequentially (in one job) + - Want to read/write simultaneously + - Key is MediaType -- it must match + Passed to SD as a sort of BSR record called Storage Specification Record or SSR. SSR @@ -256,7 +377,6 @@ For version 1.35: MediaType -> Next MediaType Pool -> Next Pool Device -> Next Device - Write Copy Resource that makes a copy of a resource. Job Resource Allow multiple Storage specifications New flags @@ -268,8 +388,8 @@ For version 1.35: Storage Allow Multiple Pool specifications (note, Pool currently in Job resource). - Allow Multiple MediaType specifications - Allow Multiple Device specifications + Allow Multiple MediaType specifications in Dir conf + Allow Multiple Device specifications in Dir conf Perhaps keep this in a single SSR Tie a Volume to a specific device by using a MediaType that is contained in only one device. @@ -285,20 +405,10 @@ Ideas from Jerry Scharf: even more important, it's not flaky it has an open access catalog, opening many possibilities it's pushing toward heterogeneous systems capability - simple things: - I don't remember an include file directive for config files - (not filesets, actual config directives) - can you check the configs without starting the daemon? - some warnings about possible common mistakes big things: - doing the testing and blessing of concurrent backup writes - this is absolutely necessary in the enterprise - easy user recovery GUI with full access checking Macintosh file client macs are an interesting niche, but I fear a server is a rathole working bare iron recovery for windows - much better handling on running config changes - thinking through the logic of what happens to jobs in progress the option for inc/diff backups not reset on fileset revision a) use both change and inode update time against base time b) do the full catalog check (expensive but accurate) @@ -351,9 +461,7 @@ Ideas from Jerry Scharf: to the user, who would then use "mount" as described above once he had actually inserted the disk. - Implement dump/print label to UA -- Implement disk spooling. Two parts: 1. Spool to disk then - immediately to tape to speed up tape operations. 2. Spool to - disk only when the tape is full, then when a tape is hung move +- Spool to disk only when the tape is full, then when a tape is hung move it to tape. - Scratch Pool where the volumes can be re-assigned to any Pool. - bextract is sending everything to the log file ****FIXME**** @@ -670,12 +778,8 @@ Ideas from Jerry Scharf: if full status requested or if some level of debug on. - Make database type selectable by .conf files i.e. at runtime - Set flag for uname -a. Add to Volume label. -- Implement throttled work queue. - Restore files modified after date - SET LD_RUN_PATH=$HOME/mysql/lib/mysql -- Implement Restore FileSet= -- Create a protocol.h and protocol.c where all protocol messages - are concentrated. - Remove duplicate fields from jcr (e.g. jcr.level and jcr.jr.Level, ...). - Timout a job or terminate if link goes down, or reopen link and query. - Concept of precious tapes (cannot be reused). @@ -694,37 +798,6 @@ Ideas from Jerry Scharf: > woorkstations to be shut down overnight to save power. > -- From Terry Manderson - jobdefs { # new structure - name = "monthlyUnixBoxen" - type = backup - level = full - schedule = monthly - storage = DLT - messages = Standard - pool = MonthlyPool - priority = 10 - } - - job { - name = "wakame" - jobdefs = "genericUnixSet" - client = wakame-fd - } - - job { - name = "durian" - jobdefs = "genericUnixSet" - client = durian-fd - } - - job { - name = "soy" - jobdefs = "UnixDevelBoxSet" - client = soy-fd - } - - - Autolabel should be specified by DIR instead of SD. - Storage daemon - Add media capacity @@ -784,6 +857,105 @@ Ideas from Jerry Scharf: - Store info on each file system type (probably in the job header on tape. This could be the output of df; or perhaps some sort of /etc/mtab record. +========= ideas =============== +From: "Jerry K. Schieffer" +To: +Subject: RE: [Bacula-users] future large programming jobs +Date: Thu, 26 Feb 2004 11:34:54 -0600 + +I noticed the subject thread and thought I would offer the following +merely as sources of ideas, i.e. something to think about, not even as +strong as a request. In my former life (before retiring) I often +dealt with backups and storage management issues/products as a +developer and as a consultant. I am currently migrating my personal +network from amanda to bacula specifically because of the ability to +cross media boundaries during storing backups. +Are you familiar with the commercial product called ADSM (I think IBM +now sells it under the Tivoli label)? It has a couple of interesting +ideas that may apply to the following topics. + +1. Migration: Consider that when you need to restore a system, there +may be pressure to hurry. If all the information for a single client +can eventually end up on the same media (and in chronological order), +the restore is facillitated by not having to search past information +from other clients. ADSM has the concept of "client affinity" that +may be associated with it's storage pools. It seems to me that this +concept (as an optional feature) might fit in your architecture for +migration. + +ADSM also has the concept of defining one or more storage pools as +"copy pools" (almost mirrors, but only in the sense of contents). +These pools provide the ability to have duplicte data stored both +onsite and offsite. The copy process can be scheduled to be handled +by their storage manager during periods when there is no backup +activity. Again, the migration process might be a place to consider +implementing something like this. + +> +> It strikes me that it would be very nice to be able to do things +like +> have the Job(s) backing up the machines run, and once they have all +> completed, start a migration job to copy the data from disks Volumes +to +> a tape library and then to offsite storage. Maybe this can already +be +> done with some careful scheduling and Job prioritzation; the events +> mechanism described below would probably make it very easy. + +This is the goal. In the first step (before events), you simply +schedule +the Migration to tape later. + +2. Base jobs: In ADSM, each copy of each stored file is tracked in +the database. Once a file (unique by path and metadata such as dates, +size, ownership, etc.) is in a copy pool, no more copies are made. In +other words, when you start ADSM, it begins like your concept of a +base job. After that it is in the "incremental" mode. You can +configure the number of "generations" of files to be retained, plus a +retention date after which even old generations are purged. The +database tracks the contents of media and projects the percentage of +each volume that is valid. When the valid content of a volume drops +below a configured percentage, the valid data are migrated to another +volume and the old volume is marked as empty. Note, this requires +ADSM to have an idea of the contents of a client, i.e. marking the +database when an existing file was deleted, but this would solve your +issue of restoring a client without restoring deleted files. + +This is pretty far from what bacula now does, but if you are going to +rip things up for Base jobs,..... +Also, the benefits of this are huge for very large shops, especially +with media robots, but are a pain for shops with manual media +mounting. + +> +> Base jobs sound pretty useful, but I'm not dying for them. + +Nobody is dying for them, but when you see what it does, you will die +without it. + +3. Restoring deleted files: Since I think my comments in (2) above +have low probability of implementation, I'll also suggest that you +could approach the issue of deleted files by a mechanism of having the +fd report to the dir, a list of all files on the client for every +backup job. The dir could note in the database entry for each file +the date that the file was seen. Then if a restore as of date X takes +place, only files that exist from before X until after X would be +restored. Probably the major cost here is the extra date container in +each row of the files table. + +Thanks for "listening". I hope some of this helps. If you want to +contact me, please send me an email - I read some but not all of the +mailing list traffic and might miss a reply there. + +Please accept my compliments for bacula. It is doing a great job for +me!! I sympathize with you in the need to wrestle with excelence in +execution vs. excelence in feature inclusion. + +Regards, +Jerry Schieffer + +============================== + Longer term to do: - Design at hierarchial storage for Bacula. Migration and Clone. - Implement FSM (File System Modules). @@ -873,177 +1045,6 @@ Need: VolSessionId and VolSessionTime. ========================================================= - - -============================================================= - - Request For Comments For File Backup Options - 10 November 2002 - -Subject: File Backup Options - -Problem: - A few days ago, a Bacula user who is backing up to file volumes and - using compression asked if it was possible to suppress compressing - all .gz files since it was a waste of CPU time. Although Bacula - currently permits using different options (compression, ...) on - a directory by directory basis, it cannot do it on a file by - file basis, which is clearly what was desired. - -Proposed Implementation: - To solve this problem, I propose the following: - - - Add a new Director resource type called Options. - - - The Options resource will have records for all - options that can currently be specified on the Include record - (in a FileSet). Examples below. - - - The Options resource will permit an exclude option as well - as a number of additional options. - - - The heart of the Options resource is the ability to - supply any number of Match records which specify POSIX - regular expressions. These Match regular expressions are - applied to the fully qualified filename (path and all). If - one matches, then the Options will be used. - - - When an Match specification matches an included file, the - options specified in the Options resource will override - the default options specified on the Include record. - - - Include records will be modified to permit referencing one or - more Options resources. The Options will be used - in the order listed on the Include record and the first - one that matches will be applied. - - - Options (or specifications) currently supplied on the Include - record will be deprecated (i.e. removed in a later version a - year or so from now). - - - The Exclude record will be deprecated as the same functionality - can be obtained by using an Exclude = yes in the Options. - -Options records: - The following records can appear in the Options resource. An - asterisk preceding the name indicates a feature not currently - implemented. - - - Regexp "xxx" - Match regular expression - - Wild "xxx" - Do a wild card match - - For Backup Jobs: - - Compression= (GZIP, ...) - - Signature= (MD5, SHA1, ...) - - *Encryption= - - OneFs= (yes/no) - remain on one filesystem - - Recurse= (yes/no) - recurse into subdirectories - - Sparse= (yes/no) - do sparse file backup - - *Exclude= (yes/no) - exclude file from being saved - - *Reader= (filename) - external read (backup) program - - *Plugin= (filename) - read/write plugin module - - - Include= (yes/no) - Include the file matched no additional - patterns are applied. - - For Verify Jobs: - - verify= (ipnougsamc5) - verify options - - For Restore Jobs: - - replace= (always/ifnewer/ifolder/never) - replace options currently - implemented in 1.31 - - *Writer= (filename) - external write (restore) program - - -Implementation: - Currently options specifying compression, MD5 signatures, recursion, - ... of a FileSet are supplied on the Include record. These will now - all be collected into a Options resource, which will be - specified in the Include in place of the options. Multiple Options - may be specified. Since the Options may contain regular expressions - that are applied to the full filename, this will give the ability - to specify backup options on a file by file basis to whatever level - of detail you wish. - -Example: - - Today: - - FileSet { - Name = "FullSet" - Include = compression=GZIP signature=MD5 { - / - } - } - - Proposal: - - FileSet { - Name = "FullSet" - Include { - Compression = GZIP; - Signature = MD5 - Wild = /*.?*/ # matches all files. - File = / - } - } - - That's a lot more to do the same thing, but it gives the ability to - apply options on a file by file basis. For example, suppose you - want to compress all files but not any file with extensions .gz or .Z. - In that case, you will need to group two sets of options using - the Options resource. Files may be anywhere except in an - option set??? All OptionSets apply to all files in the order - the OptionSets were specified. To have files included with - different option sets without using wild-cards, use two or more - Includes -- each one is handled in turn using only the files and - optionsets specified in the include. - - FileSet { - Name = "FullSet" - Include { - OptionSet { - Signature = MD5 - # Note multiple Matches are ORed - Wild = "*.gz" # matches .gz files - Wild = "*.Z" # matches .Z files - } - OptionSet { - Compression = GZIP - Signature = MD5 - Wild = "*.?*" # matches all files - } - File = / - } - } - - Now, since the no Compression option is specified in the - first group of Options, *.gz or *.Z file will have an MD5 signature computed, - but will not be compressed. For all other files, the *.gz *.Z will not - match, so the second group of options will be used which will include GZIP - compression. - -Questions: - - Is it necessary to provide some means of ANDing regular expressions - and negation? (not currently planned) - - e.g. Wild = /*.gz/ && !/big.gz/ - - - I see that Networker has a "null" module which, if specified, does not - backup the file, but does make an record of the file in the catalog - so that the catalog will reflect an exact picture of the filesystem. - The result is that the file can be "seen" when "browsing" the save - sets, but it cannot be restored. - - Is this really useful? Should it be implemented in Bacula? - -Results: - After implementing the above, the user will be able to specify - on a file by file basis (using regular expressions) what options are - applied for the backup. - - -============================================= ========================================================== Unsaved File design @@ -1211,203 +1212,152 @@ Block Position: 0 ==== -=== Done in 1.33 -- Change console to bconsole. -- Change smtp to bsmtp. -- Fix time difference problem between Bacula and Client - so that everything is in GMT. -- Fix TimeZone problem! -- Mount a tape that is not right for the job (wrong # files on tape) - Bacula asks for another tape, fix problems with first tape and - say "mount". All works OK, but status shows: - Device /dev/nst0 open but no Bacula volume is mounted. - Total Bytes=1,153,820,213 Blocks=17,888 Bytes/block=64,502 - Positioned at File=9 Block=3,951 - Full Backup job Rufus.2003-10-26_16.45.31 using Volume "DLT-24Oct03" on device /dev/nst0 - Files=21,003 Bytes=253,954,408 Bytes/sec=2,919,016 - FDReadSeqNo=192,134 in_msg=129830 out_msg=5 fd=7 -- Upgrade to cygwin 1.5 -- Optimize fsf not to read. -- Use ioctl() fsf if it exists. Figure out where we are from - the mt_status command. Use slow fsf only if other does not work. -- Enhance "update slots" to include a "scan" feature - scan 1; scan 1-5; scan 1,2,4 ... to update the catalog -- Allow a slot or range of slots on the label barcodes command. -- Finish implementation of Verify=DiskToCatalog -- Make sure that Volumes are recycled based on "Least recently used" - rather than lowest MediaId. -- Add flag to write only one EOF mark on the tape. -- Implement autochanger testing in btape "test" command. -- Implement lmark to list everyfile marked. -- Make mark/unmark report how many files marked/unmarked. -- Keep last 5 or 10 completed jobs and show them in a similar list. -- Make a Running Jobs: output similar to current Scheduled Jobs: -- Change "create_media_record in bscan to use Archive instead of Full. -- Have some way to estimate the restore size or have it printed. -- Volume problems occurs if you have valid volume, written, then it is - truncated. You get 12-Nov-2003 11:48 rufus-sd: kernsave.2003-11-12_11.48.09 Warning: mount.c:228 Volume on /tmp is not a Bacula labeled Volume, because: - block.c:640 Read zero bytes on device /tmp. -- Make sure that 64 bit I/O packets are used on Cygwin. -- Add to supported autochangers - OS             : FreeBSD-4.9 - Auto-Changer    : QUALSTAR TLS-4210 -   Manufufactur  : Qualstar -   Tapes         : 12 (AIT1: 36GB, AIT2: 50GB all uncompressed) -   Drives        : 2xAIT2 (installed in the Qualstar: SONY SDX-500C AIT2) -- Document estimate command in tree. -- Document lsmark command in tree. -- Setup a standard job that builds a bootstrap file and saves - it with the catalog database. -- See if a restore job can add a file to the tape (prohibit this). -- Restrict characters permitted in a name. -- In restore, provide option for limiting to a particular Pool. -- In restore, list FileSets that only have different base names -- - i.e. any FileSet with the same name should be treated as the same. -- Make Scheduler sort jobs by StartTime, Priority. -- Make sure smtp and any other useful program is executable by the world - in case Bacula is not running as root. -- Look at Dan's field width problems in PostgreSQL. -- Look at effect of removing GROUP BYs. -- In restore take all filesets with same base name. -- From Alan Brown - BTW, there's a make install bug in 1.33 - with --enable-gnome, - gnome-console is built, but the binary and .conf are not being installed. -- Permit Bacula and apcupsd donations (not done for apcupsd). -- Fix Ctl-C crashing the Console (readline?). -- Look at code in recycle_oldes_purged_volume() recycle.c. Why not - let SQL do ORDER BY LastWritten ASC? -- Look at find_next_volume() algorithm. Currently, it selects: - +---------+------------+---------------------+-----------+ - | MediaId | VolumeName | LastWritten | VolBytes | - +---------+------------+---------------------+-----------+ - | 3 | Test13 | 0000-00-00 00:00:00 | 1 | - | 4 | Test14 | 0000-00-00 00:00:00 | 1 | - | 1 | test11 | 2003-12-03 18:39:55 | 4,004,926 | - | 2 | test12 | 2004-01-04 15:25:56 | 2,078,691 | - +---------+------------+---------------------+-----------+ - but perhaps it should fill already used Volumes first, and use - Append volumes before Purged, or Recycled, ... -- Possibly remove the "|| ap == NULL" on lines 123 and 207 of lib/var.c, - which creates compile problems on alpha systems. - var.c:123: no match for `va_list & == long int' -- Check "restore" 3 (JobId), then it asks for Storage resource. Does - it verify that the correct volume is chosen? -- Make Bacula "poll a drive". -- Notes for final checking of Nic's code: - Could I get you to double check the switch () statements in the - job_check_maxwaittime and job_check_maxruntime functions in - src/dird/job.c? -- Define week of year for scheduler. W01, W02, ... - Week 01 of a year is per definition the first week that has the - Thursday in this year, which is equivalent to the week that contains the - fourth day of January. In other words, the first week of a new year is - the week that has the majority of its days in the new year. Week 01 - might also contain days from the previous year and the week before week - 01 of a year is the last week (52 or 53) of the previous year even if it - contains days from the new year. A week starts with Monday (day 1) and - ends with Sunday (day 7). For example, the first week of the year 1997 - lasts from 1996-12-30 to 1997-01-05 and can be written in standard - notation as - 1997-W01 or 1997W01 - The week notation can also be extended by a number indicating the day - of the week. For example, the day 1996-12-31, which is the Tuesday (day - 2) of the first week of 1997, can also be written as - 1997-W01-2 or 1997W012 -- Either restrict the characters in a name, or fix the problem - emailing with names containing / (smtp command line breaks). -- Implement .consolerc for Console -- Implement scan: for every slot it finds, zero the slot of - Volume other volume having that slot. -- Make restore job check if all the files are actually restored. -- Look at 2Gb limit for SQLite. -- Fix get_storage_from_media_type (ua_restore) to use command line - storage= -- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume. -- Write a mini-readline with history and editing. -- Take a careful look a the Basic recycling algorithm. When Bacula - chooses, the order should be: - - Look for Append - - Look for Recycle or Purged - - Prune volumes - - Look for purged - Instead of using lowest media Id, find the least recently used - volume. - - When the tape is mounted and Bacula requests the status - - Do everything possible to use it. - - Define a "available" status, which is the currently mounted - Volume and all volumes that are currently in the autochanger. -- Is a pool specification really needed for a restore? Yes, and - you may want to exclude archive Pools. -- Implement a PostgreSQL driver. -- Fix restore to list errors if Invalid block found, and if # files - restored does not match # expected. -- Something is not right in last block of fill command. -- Add FileSet to command line arguments for restore. -- Enhance time and size scanning routines. -- Add Console usr permissions -- do by adding filters for - jobs, clients, storage, ... -- Put max network buffer size on a directive. -- Why does "mark cygwin" take so long!!!!!!!! -- Implement alist processing for ACLs from Console. -- When a file is set for restore, walk back up the chain of - directories, setting them to be restored. -- Figure out a way to set restore on a directory without recursively - decending. (recurse off?). -- Fix restore to only pull in last Differential and later Incrementals. -- Implement 3 Pools for a Job: - Job { -   Name = ... -   Full Backup Pool = xxx -   Incremental Backup Pool = yyy -   Differential Backup Pool = zzz - } -- Look at ASSERT() at 384 src/lib/bnet.c -- Dates are wrong in restore list from Win32 FD. -- Dates are wrong in catalog from Win32 FD. -- Remove h_errno from bnet.c by including proper header. -- For "list jobs" order by EndTime. -- Make two tape fill test work. -- Add atime preservation. -- Do not err job if could not write bootstrap file. -- Save and restore last_job across executions. -- Have each daemon save the last_jobs structure when exiting and - read it back in when starting up. -- "restore jobid=1 select" calls get_storage_xxx, which prints "JobId 1 is - not running." -- Make column listing for running jobs JobId Level Type Started Name Status -- Why does Bacula need the drive open to do "autochanger list" ? -- Add data compare on write/read in btape "test". -- Rescue builds incorrect script files on Rufus. -- Release SQLite 2.8.9 -- During install, copy any console.conf to bconsole.conf. -- Check: Run = Level=Differential feb-dec 1 at 1:05 to see if wday is empty. -- Look at installation file permissions with Scott so that make install - and the rpms agree. -- Finish code passing files=nnn to restore start. -- Add ctl-c to console to stop current command and discard buffered - output. -- Estimate to Tibs never returns. -- Symbolic link a directory to another one, then backup the symbolic link. -- Check and possibly fix problems with hard links. -- Fix query buffer clobber ua_query.c -- Allow "delete job jobid=xx jobid=xxx". -- Update volume FromPool (or FromPool=xxx) refreshes the Volume defaults from Pool. -- Implement multiple Volume in "purge jobs volume=". -- Test Qmsg() code to be used in bnet.c to prevent recursion. Queue the - message. If dequeueing toss the messages. Lock while dequeuing so that - it cannot be called recursively and set dequeuing flag. -- On unknown client in restore "client=xxx" - Could not find Client "Matou": ERR=Query failed: DROP TABLE temp1: - ERR=no such table: temp1 -- Null dlist head and tail pointers if all items are removed. -- Build console in client-only build. -- Phil says that Windows file sizes mismatch in Verify when they should, - and that either the file size or the catalog size was zero. -- Check time/dates printed during restore when using Win32 API. -- Once a job is canceled, make sure the status reflects that even if the - job is "stuck" in the run queue. -- Test work on conio.c -- particularly linking. -- Complete Win32 installer -- Add ms disk example +=== Done in 1.34.3 +- To pass Include 1 or two letter commands + I Name Include name - first record + B Name Base name - repeat + R "xxx" Regexp + W "xxx" Wild Card + E zzz Exclude expression (wild card) + P "plugin" Plugin + D "reader" Reader program + T "writer" Writer program + O Options In current commpressed format + (compression, signature, onefs, recurse, sparse, + replace, verify options, ...) + N End option set + B BaseName Start second option set any letter + ... + E + F Number Number of filenames to follow + + B Name + ... + N End option set + F Number Number of filenames to follow + + ... +- Add regression of btape "fill" +==== Done in 1.34.4 +- Implement "label slots=1,2... pool=yyy barcodes" +- Mark a non-removable disk Volume in error rather than asking sys op to + mount it. +- Implement Fixed storage LabelFormat test. +- Add reporting in attr despooling. +- Figure out how to use ssh or stunnel to protect Bacula communications. + Add Dan's work to manual See ssl.txt + +==== Done in 1.35.0 +- Implement fast tree insert (doubly linked list) +- Disallow using Internal database +- Feedback while the restore tree is being built. +- Add detection of Whitebox release in ./configure +- Add InChanger to "list media" +- Implement second cut of SIGHUP in Dir + (Implement resources on a single pointer) +- Add regular expressions to FileSets (needs testing). +- Test new despooling error recovery code when partition fills. +- Sort Scheduled jobs status listing by start time. +- Add priority to Scheduled jobs status listing. +- Look at Win32 inc problem when new directory added files not saved??? +- Require resource names to be unique. +- Make sure that all errors from libacl are printed. +- Make FD run on Win95 if possible (not really possible). +- Fix GROUP BYs in all SQL (done by Dan -- thanks) +- Make entering multiple Storage names in Dir illegal. +- Make btape release the drive during the "test" append. +- Fix restore: + You have selected the following JobIds: 739,856,797 + Building directory tree for JobId 739 ... + Building directory tree for JobId 856 ... + Building directory tree for JobId 797 ...3 + Jobs inserted into the tree. +- Fix errno handling in win32 compat routines. +- Add better error codes to run_program (10000+) +- Revisit and revise Disaster Recovery (fix SCSI and RAID + disk detection) +- When passwords do not match, print message that points the + user to the doc. +- Do tape alerts -- see tapealert.txt +- Find memory leaks in address_conf.c +- Network order of port and ip addresses. +- SetIP in dird/fd_cmds.c is broken. +- don't use inet_addr() it is deprecated. +- Fix code in filed.c for init_bsock(); +- Fix code in console.c for init_bsock(); +- Solve the termcap.h problem on Solaris configure. +- Make Verify jobs require exclusive use of Volume as Restore + jobs do. +- Fix restore ++++ that get intermingled with "Building directory tree" +- Fix doc in dirdconf.html for @, where the file must + have not a list of files, but + file = xxx + file = yyy +- Test Win32 errno handling. +- port 1.34.6-block.patch to 1.35 +- Look at Chris' patch for bscan bug. +- Fix error handling in spooling both data and attribute. +- Implement Ignore FileSet Change. +- Doc new duration time input editing. +- Bacula rescue CDROM implement isolinux +- Make sure Qmsgs are dequeued by FD and SD. +- Check if ACLs allocated at dird_conf.c:1214 are being properly + released. +- Test/doc Tape Alerts +- Doc dbcheck eliminate orphaned clients. +- Doc Phil's new delete job jobid scanning code. +- Document that console commands can be abbreviated. +- Doc update AllFromVol +- Doc -p option in stored +- New IP address specification is used as follows: + [sdaddresses|diraddresses|fdaddresses] = { [[ip|ipv4|ipv6] = { + [[addr|port] = [^ ]+[\n;]+] }] } + + so it could look for example like this: + SDaddresses = { ip = { + addr = 1.2.3.4; port = 1205; } + ipv4 = { + addr = 1.2.3.4; port = http; } + ipv6 = { + addr = 1.2.3.4; + port = 1205; + } + ip = { + addr = 1.2.3.4 + port = 1205 + } + ip = { + addr = 1.2.3.4 + } + ip = { + addr = 201:220:222::2 + } + ip = { + addr = bluedot.thun.net + } + } + as a consequence, you can now specify multiple IP addresses and + ports to be used. In the case of a server, it will listen on + all those that you specify. In the case of connecting to the server, + Bacula will attempt connecting to one at a time until it succeeds. + And, in a few other special cases, Bacula will use only the first + address specified. + + The default port numbers are still the same and the services and hosts + are also resolved by name. So now you could use the real names for the + port numbers. + + An ip section will allow resolution to either an ipv4 or an ipv6 address. + An ipv4 section forces the resolution to be only ipv4, and an ipv6 section + forces the resolution to be only ipv6. +- Fix silly restriction requiring Include { Options { xxx } } to be + on separate lines. +- Restore c: with a prefix into /prefix/c/ to prevent c: and d: + files with the same name from overwritting each other. +- Add "Multiple connections = yes/no" to catalog resource. +- Add new DCR calling sequences everywhere in SD. This will permit + simultaneous use of multiple devices by a single job. +- Fix bscan so that it releases the drive when requesting a new tape. +- List verify options for DiskToCatalog in doc. +- Turn on transactions if multiple connections are on in DB. +