X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fkernstodo;h=ec70bd1f54a377a1a59a4ed70ac98ffac360a555;hb=2dab03b7fc53e23e1c9cddb8634905db4243c39c;hp=67a1263208d3b28ae04f471e07a0af7d4101009b;hpb=8282883ef13dc5d1893b63711bc9d15504188195;p=bacula%2Fbacula diff --git a/bacula/kernstodo b/bacula/kernstodo index 67a1263208..ec70bd1f54 100644 --- a/bacula/kernstodo +++ b/bacula/kernstodo @@ -1,7 +1,9 @@ Kern's ToDo List - 22 December 2003 + 27 January 2004 Documentation to do: (any release a little bit at a time) +- DB upgrade to version 5 in bacula-1.27b, DB upgrade to + version 6 in 1.31; DB upgrade to version 7 in 1.33/4. - Document running a test version. - Document query file format. - Document static linking @@ -28,12 +30,14 @@ Documentation to do: (any release a little bit at a time) - Create a man page for each binary (Debian package requirement). Testing to do: (painful) +- Test drive polling! - that ALL console command line options work and are always implemented - blocksize recognition code. - Test if rewind at end of tape waits for tape to rewind. - Test cancel at EOM. For 1.33 Testing/Documentation: +- Newly labeled tapes are chosen before ones already in use. - Document new alias records in Director. SDAddress SDDeviceName, SDPassword. FDPassword, FDAddress, DBAddress, DBPort, DBPassword. - Document new Include/Exclude ... @@ -41,6 +45,7 @@ For 1.33 Testing/Documentation: - Add counter variable test. - Document ln -sf /usr/lib/libncurses.so /usr/lib/libtermcap.so and install the esound-dev  package for compiling Console on SuSE. + This should read 'Document LDFLAGS="-L/usr/lib/termcap" ... ' - Add an example of using a FIFO in dirdconf.wml - Add an item to the FAQ about running jobs in different timezones. - Add some examples of job editing codes. @@ -53,17 +58,68 @@ For 1.33 Testing/Documentation: - Add db check test to regression. Test each function like delete, purge, ... - Add subsections to the Disaster Recovery index section. - +- Document Pool keyword for restore. +- If you use restore replace=never, the directory attributes for + non-existent directories will not be restored properly. +- In the Bacula User Guide you write:"Note, one major disadvantage of + writing to a NFS mounted volume as I do isthat if the other machine goes + down, the OS will wait forever on the fopen()call that Bacula makes. As + a consequence, Bacula will completely stall untilthe machine exporting + the NSF mounts comes back up. If someone knows a wayaround this, please + let me know."I haven't tried using NFS in years, but I think that the + "soft" and "intr"remount options may well help you. The only way of + being sure would be totry it.See, for example, + http://howtos.linux.com/guides/nag2/x-087-2-nfs.mountd.shtml + For 1.33 -- Make Bacula "poll a drive". +- Fix FreeBSD mt_count problem. +- During install, copy any console.conf to bconsole.conf. +- Have each daemon save the last_jobs structure when exiting and + read it back in when starting up. +- "restore jobid=1 select" calls get_storage_xxx, which prints "JobId 1 is + not running." +- Add FromClient and ToClient keywords on restore command (or + BackupClient RestoreClient). +- Automatic "update slots" on user configuration directive when a + slot error occurs. +- Allow "delete job jobid=xx jobid=xxx". +- Allow "delete job jobid=xxx,yyy,aaa-bbb" i.e. list + ranges. +- Implement multiple Volume in "purge jobs volume=". +- Implement a JobSet, which groups any number of jobs. If the + JobSet is started, all the jobs are started together. + Allow Pool, Level, and Schedule overrides. +- Enhance cancel to timeout BSOCK packets after a specific delay. +- When I restore to Windows the Created, Accessed and Modifiedtimes are + those of the time of the restore, not those of the originalfile. + The dates you will find in your restore log seem to be the original + creation dates +- Rescue builds incorrect script files on Rufus. +- Write a Qmsg() to be used in bnet.c to prevent recursion. Queue the + message. If dequeueing toss the messages. Lock while dequeuing so that + it cannot be called recursively and set dequeuing flag. +- Add all pools in Dir conf to DB also update them to catch changed + LabelFormats and such. +- Symbolic link a directory to another one, then backup the symbolic + link. +- Build console in client-only build. +- Restore attributes of directory if replace=never set but directory + did not exist. +- Check why Phil's Verify exclude does not work. +- Phil says that Windows file sizes mismatch in Verify when they should, + and that either the file size or the catalog size was zero. +- Fix option 2 of restore -- list where file is backed up -- require Client, + then list last 20 backups. +- Allow browsing the catalog to see all versions of a file (with + stat data on each file). +- Finish code passing files=nnn to restore start. +- Add level to estimate command. +- Check time/dates printed during restore when using Win32 API. +- Volume "add"ed to Pool gets recycled in first use. VolBytes=0 +- Get rid of 0 dates in LastWritten, ... - Fix "llist jobid=xx" where no fileset or client exists. -- Release SQLite 2.8.8 +- Release SQLite 2.8.9 - If a tape is recycled while it is mounted, Stanislav Tvrudy must do an additional mount to deblock the job. -- Notes for final checking of Nic's code: - Could I get you to double check the switch () statements in the - job_check_maxwaittime and job_check_maxruntime functions in - src/dird/job.c? - From Johan Decock: bscan: sql_update.c:65 UPDATE File SET MD5='Ij+5kwN6TFIxK+8l8+/I+A' WHERE FileId=0 bscan: bscan.c:1074 Could not add MD5/SHA1 to File record. ERR=sql_update.c:65 Update problem: affected_rows=0 @@ -101,6 +157,7 @@ For 1.33 resources, like Level? If so, I think I'd make it an optional directive in Job, Client, and Pool, with precedence such that Job overrides Client which in turn overrides Pool. +- Print a message when a job starts if the conf file is not current. - Finish work on conio.c - To pass Include 1 or two letter commands I Name Include name - first record @@ -126,6 +183,67 @@ For 1.33 F Number Number of filenames to follow ... + +- Spooling ideas taken from Volker Sauer's and other's emails: + > IMHO job spooling should be turned on + > + > 1) by job + > 2) by schedule + > 3) by sd + > + > where and 2) overrides 1) and 3) is independent. + + Yes, this is the minimum that I think is necessary. + + > + > Reason(s): + > It should be switched by job, because the job that backs up the machine + > with the bacula-sd on doesn't need spooling. + > It should be switched by schedule, because for full-backups I don't need + > spooling, so I can switch it off (because the network faster then the + > tapedrive) + + True, with the exception that if you have enough disk spool space, + and you want to run concurrent jobs, spooling can eliminate the block + interleaving restore inefficiencies. + + > And you should be able to turn it of by sd for sd-machines with low disk + > capacity or if you just don't need or want this feature. + > + > There should be: + > - definitly the possibility for multipe spool direcories + + Having multiple directories is no problem -- having different maximum + sizes creates specification problems. At some point, I will probably + have a common SD pool of spool directories as well as a set of + private spool directories for each device. The first implementation + will be a set of private spool directories for each device since + managing a global pool with a bunch of threads writing into the same + directory is *much* more complicated and prone to error. + + > - the ability to spool parts of a backup (not the whole client) + + This may change in the future, but for the moment, it will spool + either to a job high water mark, or until the directory is full + (reaches max spool size or I/O error). It will then write to tape, + truncate the spool file, and begin spooling again. + + > - spooling while writing to tape + + Not within a job, but yes, if you run concurrent jobs -- each is a + different thread. Within a job could be a feature, but *much* later. + + > - parallel spooling (like parallel jobs/ concurrent jobs) of clients + + Yes, this is one of my main motivations for doing it (aside from + eliminating tape "shoe shine" during incremental backups. + + > - flushing a backup that only went to disk (like amflush in amanda) + + This will be a future feature, since spooling is different from backing + up to disk. The future feature will be "migration" which will move a job + from one backup Volume to another. + - New Storage specifications: Passed to SD as a sort of BSR record called Storage Specification Record or SSR. @@ -154,6 +272,46 @@ For 1.33 In SD allow Device to have Multiple MediaTypes After 1.33: +- Look at www.nu2.nu/pebuilder as a helper for full windows + bare metal restore. +From Chris Hull: + it seems to be complaining about 12:00pm which should be a valid 12 + hour time. I changed the time to 11:59am and everything works fine. + Also 12:00am works fine. 0:00pm also works (which I don't think + should). None of the values 12:00pm - 12:59pm work for that matter. +Ideas from Jerry Scharf: + First let's point out some big pluses that bacula has for this + it's open source + more importantly it's active. Thank you so much for that + even more important, it's not flaky + it has an open access catalog, opening many possibilities + it's pushing toward heterogeneous systems capability + simple things: + I don't remember an include file directive for config files + (not filesets, actual config directives) + can you check the configs without starting the daemon? + some warnings about possible common mistakes + big things: + doing the testing and blessing of concurrent backup writes + this is absolutely necessary in the enterprise + easy user recovery GUI with full access checking + Macintosh file client + macs are an interesting niche, but I fear a server is a rathole + working bare iron recovery for windows + much better handling on running config changes + thinking through the logic of what happens to jobs in progress + the option for inc/diff backups not reset on fileset revision + a) use both change and inode update time against base time + b) do the full catalog check (expensive but accurate) + sizing guide (how much system is needed to back up N systems/files) + consultants on using bacula in building a disaster recovery system + an integration guide + or how to get at fancy things that one could do with bacula + logwatch code for bacula logs (or similar) + linux distro inclusion of bacula (brings good and bad, but necessary) + win2k/XP server capability (icky but you asked) + support for Oracle database ?? +=== - Look at adding SQL server and Exchange support for Windows. - Restore: Enter Filename: 'C:/Documents and Settings/Comercial/My Documents/MOP/formulário de registro BELAS ARTES.doc' causes Bacula to @@ -177,38 +335,10 @@ After 1.33: Job report (Volker Sauer). - Client does not show busy during Estimate command. - Implement Console mtx commands. -- Look at 2Gb limit for SQLite. -- Implement 3 Pools for a Job: - Job { -   Name = ... -   Full Backup Pool = xxx -   Incremental Backup Pool = yyy -   Differential Backup Pool = zzz - } - Add a default DB password to MySQL. GRANT all privileges ON bacula.* TO bacula@localhost IDENTIFIED BY 'bacula_password'; FLUSH PRIVILEGES; -- Define week of year for scheduler. W01, W02, ... - Week 01 of a year is per definition the first week that has the - Thursday in this year, which is equivalent to the week that contains the - fourth day of January. In other words, the first week of a new year is - the week that has the majority of its days in the new year. Week 01 - might also contain days from the previous year and the week before week - 01 of a year is the last week (52 or 53) of the previous year even if it - contains days from the new year. A week starts with Monday (day 1) and - ends with Sunday (day 7). For example, the first week of the year 1997 - lasts from 1996-12-30 to 1997-01-05 and can be written in standard - notation as - - 1997-W01 or 1997W01 - - The week notation can also be extended by a number indicating the day - of the week. For example, the day 1996-12-31, which is the Tuesday (day - 2) of the first week of 1997, can also be written as - - 1997-W01-2 or 1997W012 - - Implement a Mount Command and an Unmount Command where the users could specify a system command to be performed to do the mount, after which Bacula could attempt to @@ -262,7 +392,6 @@ After 1.33: - Take a careful look at Level for the estimate command, maybe make it a command line option. - Add Volume name to "I cannot write on this volume because" -- Make restore job check if all the files are actually restored. - Make tree walk routines like cd, ls, ... more user friendly by handling spaces better. - Write your PID file and chown root:wheel before drop. @@ -291,9 +420,7 @@ After 1.33: - Set IO_NOWAIT on Bacula TCP/IP packets. - Try doing a raw partition backup and restore by mounting a Windows partition. -- Report CVS problems to SourceForge. -- Implement .consolerc for Console -- From Lars Köllers: +- From Lars Kellers: Yes, it would allow to highly automatic the request for new tapes. If a tape is empty, bacula reads the barcodes (native or simulated), and if an unused tape is found, it runs the label command with all the @@ -302,8 +429,6 @@ After 1.33: By the way can bacula automatically "move" an empty/purged volume say in the "short" pool to the "long" pool if this pool runs out of volume space? -- Either restrict the characters in a name, or fix the problem - emailing with names containing / (smtp command line breaks). - Eliminate orphaned jobs: dbcheck, normal pruning, delete job command. Hm.  Well, there are the remaining orphaned job records: @@ -352,14 +477,9 @@ After 1.33: - Walk through the Pool records rather than the Job records in dird.c to create/update pools. - What to do about "list files job=xxx". -- Implement scan: for every slot it finds, zero the slot of - Volume other volume having that slot. - When job rescheduled, status gives is waiting for Client Rufus to connect to Storage File. Dir needs to inform SD that job is rescheduled. -- Fix get_storage_from_media_type (ua_restore) to use command line - storage= -- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume. - Make Dmsg look at global before calling subroutine. - Enable trace output at runtime for Win32 - Available volumes for autochangers (see patrick@baanboard.com 3 Sep 03 @@ -367,7 +487,6 @@ After 1.33: - Get and test MySQL 4.0 - Do a complete audit of all pthreads_mutex, cond, ... to ensure that any that are dynamically initialized are destroyed when no longer used. -- Write a mini-readline with history and editing. - Look at how fuser works and /proc/PID/fd that is how Nic found the file descriptor leak in Bacula. - Implement WrapCounters in Counters. @@ -378,24 +497,7 @@ After 1.33: - Can we dynamically change FileSets? - If pool specified to label command and Label Format is specified, automatically generate the Volume name. -- Take a careful look a the Basic recycling algorithm. When Bacula - chooses, the order should be: - - Look for Append - - Look for Recycle or Purged - - Prune volumes - - Look for purged - Instead of using lowest media Id, find the least recently used - volume. - - When the tape is mounted and Bacula requests the status - - Do everything possible to use it. - - Define a "available" status, which is the currently mounted - Volume and all volumes that are currently in the autochanger. - - Why can't SQL do the filename sort for restore? -- Is a pool specification really needed for a restore? Yes, and - you may want to exclude archive Pools. - Look at libkse (man kse) for FreeBSD threading. - Look into Microsoft Volume Shadowcopy Service VSS for backing up system state components (Active Directory, System Volume, ...) @@ -405,10 +507,7 @@ After 1.33: time as the user walks through the tree). - Possibly use the hash code if the user selects all for a restore command. - Orphaned Dir buffer at parse_conf.c:373 => store_dir -- Add Console usr permissions. - Fix "restore all" to bypass building the tree. -- Fix restore to list errors if Invalid block found, and if # files - restored does not match # expected. - Prohibit backing up archive device (findlib/find_one.c:128) - Implement Release Device in the Job resource to unmount a drive. - Implement Acquire Device in the Job resource to mount a drive, @@ -419,7 +518,6 @@ After 1.33: - Make things like list where a file is saved case independent for Windows. - Implement migrate -- Implement a PostgreSQL driver. - Bacula needs to propagate SD errors. > > cluster-dir: Start Backup JobId 252, Job=REUTERS.2003-08-11_15.04.12 > > prod4-sd: REUTERS.2003-08-11_15.04.12 Error: Write error on device @@ -440,9 +538,7 @@ After 1.33: We must cd into the directory then create the file without the full path name. - lstat() is not going to work on Win32 for testing date. -- Something is not right in last block of fill command. - Implement a Recycle command -- Add FileSet to command line arguments for restore. - Add client name to cram-md5 challenge so Director can immediately verify if it is the correct client. - Add JobLevel in FD status (but make sure it is defined). @@ -501,7 +597,6 @@ After 1.33: - Implement restore "current system", but take all files without doing selection tree -- so that jobs without File records can be restored. -- Implement a relocatable bacula.spec - Add prefixlinks to where or not where absolute links to FD. - Issue message to mount a new tape before the rewind. - Simplified client job initiation for portables. @@ -552,7 +647,6 @@ After 1.33: - Add UA rc and history files. - put termcap (used by console) in ./configure and allow -with-termcap-dir. -- Enhance time and size scanning routines. - Fix Autoprune for Volumes to respect need for full save. - Fix Win32 config file definition name on /install - Compare tape to Client files (attributes, or attributes and data) @@ -787,7 +881,7 @@ Need: ============================================================= - Request For Comments For File Backup Options + Request For Comments For File Backup Options 10 November 2002 Subject: File Backup Options @@ -993,6 +1087,7 @@ Done: (see kernsdone for more) - Implement ClientRunBeforeJob and ClientRunAfterJob. - Implement forward spacing block/file: position_device(bsr) -- just before read_block_from_device(); + === for 1.33 - Change console to bconsole. - Change smtp to bsmtp. @@ -1056,4 +1151,93 @@ Done: (see kernsdone for more) gnome-console is built, but the binary and .conf are not being installed. - Permit Bacula and apcupsd donations (not done for apcupsd). - Fix Ctl-C crashing the Console (readline?). +- Look at code in recycle_oldes_purged_volume() recycle.c. Why not + let SQL do ORDER BY LastWritten ASC? +- Look at find_next_volume() algorithm. Currently, it selects: + +---------+------------+---------------------+-----------+ + | MediaId | VolumeName | LastWritten | VolBytes | + +---------+------------+---------------------+-----------+ + | 3 | Test13 | 0000-00-00 00:00:00 | 1 | + | 4 | Test14 | 0000-00-00 00:00:00 | 1 | + | 1 | test11 | 2003-12-03 18:39:55 | 4,004,926 | + | 2 | test12 | 2004-01-04 15:25:56 | 2,078,691 | + +---------+------------+---------------------+-----------+ + but perhaps it should fill already used Volumes first, and use + Append volumes before Purged, or Recycled, ... +- Possibly remove the "|| ap == NULL" on lines 123 and 207 of lib/var.c, + which creates compile problems on alpha systems. + var.c:123: no match for `va_list & == long int' +- Check "restore" 3 (JobId), then it asks for Storage resource. Does + it verify that the correct volume is chosen? +- Make Bacula "poll a drive". +- Notes for final checking of Nic's code: + Could I get you to double check the switch () statements in the + job_check_maxwaittime and job_check_maxruntime functions in + src/dird/job.c? +- Define week of year for scheduler. W01, W02, ... + Week 01 of a year is per definition the first week that has the + Thursday in this year, which is equivalent to the week that contains the + fourth day of January. In other words, the first week of a new year is + the week that has the majority of its days in the new year. Week 01 + might also contain days from the previous year and the week before week + 01 of a year is the last week (52 or 53) of the previous year even if it + contains days from the new year. A week starts with Monday (day 1) and + ends with Sunday (day 7). For example, the first week of the year 1997 + lasts from 1996-12-30 to 1997-01-05 and can be written in standard + notation as + 1997-W01 or 1997W01 + The week notation can also be extended by a number indicating the day + of the week. For example, the day 1996-12-31, which is the Tuesday (day + 2) of the first week of 1997, can also be written as + 1997-W01-2 or 1997W012 +- Either restrict the characters in a name, or fix the problem + emailing with names containing / (smtp command line breaks). +- Implement .consolerc for Console +- Implement scan: for every slot it finds, zero the slot of + Volume other volume having that slot. +- Make restore job check if all the files are actually restored. +- Look at 2Gb limit for SQLite. +- Fix get_storage_from_media_type (ua_restore) to use command line + storage= +- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume. +- Write a mini-readline with history and editing. +- Take a careful look a the Basic recycling algorithm. When Bacula + chooses, the order should be: + - Look for Append + - Look for Recycle or Purged + - Prune volumes + - Look for purged + Instead of using lowest media Id, find the least recently used + volume. + + When the tape is mounted and Bacula requests the status + - Do everything possible to use it. + Define a "available" status, which is the currently mounted + Volume and all volumes that are currently in the autochanger. +- Is a pool specification really needed for a restore? Yes, and + you may want to exclude archive Pools. +- Implement a PostgreSQL driver. +- Fix restore to list errors if Invalid block found, and if # files + restored does not match # expected. +- Something is not right in last block of fill command. +- Add FileSet to command line arguments for restore. +- Enhance time and size scanning routines. +- Add Console usr permissions -- do by adding filters for + jobs, clients, storage, ... +- Put max network buffer size on a directive. +- Why does "mark cygwin" take so long!!!!!!!! +- Implement alist processing for ACLs from Console. +- When a file is set for restore, walk back up the chain of + directories, setting them to be restored. +- Figure out a way to set restore on a directory without recursively + decending. (recurse off?). +- Fix restore to only pull in last Differential and later Incrementals. +- Implement 3 Pools for a Job: + Job { +   Name = ... +   Full Backup Pool = xxx +   Incremental Backup Pool = yyy +   Differential Backup Pool = zzz + } +- Look at ASSERT() at 384 src/lib/bnet.c