X-Git-Url: https://git.sur5r.net/?a=blobdiff_plain;f=bacula%2Fkernstodo;h=b5919192317c76bf8052f161b0bd402b8b72f581;hb=ec68e2f196aebcc44e6610fbffab25f85af3417e;hp=2daed48cbdf35a42c38bc0da9ce25663bf94deb0;hpb=1c021ed228e88e6d659fa4e443bfa50f66141220;p=bacula%2Fbacula diff --git a/bacula/kernstodo b/bacula/kernstodo index 2daed48cbd..b591919231 100644 --- a/bacula/kernstodo +++ b/bacula/kernstodo @@ -1,5 +1,5 @@ Kern's ToDo List - 17 December 2003 + 27 January 2004 Documentation to do: (any release a little bit at a time) - Document running a test version. @@ -28,12 +28,14 @@ Documentation to do: (any release a little bit at a time) - Create a man page for each binary (Debian package requirement). Testing to do: (painful) +- Test drive polling! - that ALL console command line options work and are always implemented - blocksize recognition code. - Test if rewind at end of tape waits for tape to rewind. - Test cancel at EOM. For 1.33 Testing/Documentation: +- Newly labeled tapes are chosen before ones already in use. - Document new alias records in Director. SDAddress SDDeviceName, SDPassword. FDPassword, FDAddress, DBAddress, DBPort, DBPassword. - Document new Include/Exclude ... @@ -41,34 +43,74 @@ For 1.33 Testing/Documentation: - Add counter variable test. - Document ln -sf /usr/lib/libncurses.so /usr/lib/libtermcap.so and install the esound-dev  package for compiling Console on SuSE. + This should read 'Document LDFLAGS="-L/usr/lib/termcap" ... ' - Add an example of using a FIFO in dirdconf.wml - Add an item to the FAQ about running jobs in different timezones. - Add some examples of job editing codes. - Document Dan's new --with-dir-user, ... options. + See userid.txt - Figure out how to use ssh or stunnel to protect Bacula communications. Add Dan's work to manual + See ssl.txt - Add db check test to regression. Test each function like delete, purge, ... - Add subsections to the Disaster Recovery index section. - +- Document Pool keyword for restore. +- If you use restore replace=never, the directory attributes for + non-existent directories will not be restored properly. +- In the Bacula User Guide you write:"Note, one major disadvantage of + writing to a NFS mounted volume as I do isthat if the other machine goes + down, the OS will wait forever on the fopen()call that Bacula makes. As + a consequence, Bacula will completely stall untilthe machine exporting + the NSF mounts comes back up. If someone knows a wayaround this, please + let me know."I haven't tried using NFS in years, but I think that the + "soft" and "intr"remount options may well help you. The only way of + being sure would be totry it.See, for example, + http://howtos.linux.com/guides/nag2/x-087-2-nfs.mountd.shtml + For 1.33 -- From Alan Brown - BTW, there's a make install bug in 1.33 - with --enable-gnome, - gnome-console is built, but the binary and .conf are not being installed. -- Make Scheduler sort jobs by StartTime, Priority. -- Make sure smtp and any other useful program is executable by the world - in case Bacula is not running as root. +- Rescue builds incorrect script files on Rufus. +- Write a Qmsg() to be used in bnet.c to prevent recursion. Queue the + message. If dequeueing toss the messages. Lock while dequeuing so that + it cannot be called recursively and set dequeuing flag. +- Look at ASSERT() at 384 src/lib/bnet.c +- Add all pools in Dir conf to DB. +- Symbolic link a directory to another one, then backup the symbolic + link. +- Build console in client-only build. +- Restore attributes of directory if replace=never set but directory + did not exist. +- Check why Phil's Verify exclude does not work. +- Phil says that Windows file sizes mismatch in Verify when they should, + and that either the file size or the catalog size was zero. +- Fix option 2 of restore -- list where file is backed up -- require Client, + then list last 20 backups. +- Allow browsing the catalog to see all versions of a file (with + stat data on each file). +- Finish code passing files=nnn to restore start. +- Add level to estimate command. +- Check time/dates printed during restore when using Win32 API. +- Volume "add"ed to Pool gets recycled in first use. VolBytes=0 +- Get rid of 0 dates in LastWritten, ... +- Fix "llist jobid=xx" where no fileset or client exists. +- Release SQLite 2.8.9 +- If a tape is recycled while it is mounted, Stanislav Tvrudy must do an + additional mount to deblock the job. +- From Johan Decock: + bscan: sql_update.c:65 UPDATE File SET MD5='Ij+5kwN6TFIxK+8l8+/I+A' WHERE FileId=0 + bscan: bscan.c:1074 Could not add MD5/SHA1 to File record. ERR=sql_update.c:65 Update problem: affected_rows=0 +- Do scheduling by UTC using gmtime_r() in run_conf, scheduler, and + ua_status.!!! Thanks to Alan Brown for this tip. - Look at updating Volume Jobs so that Max Volume Jobs = 1 will work correctly for multiple simultaneous jobs. - Correct code so that FileSet MD5 is calculated for < and | filename generation. -- Permit Bacula and apcupsd donations. - Mark Volume in error on error from WEOF. - Why does Bacula need the drive open to do "autochanger list" ? - Add a .list all files in the restore tree (probably also a list all files) Do both a long and short form. -- Add a Media record flag that indicates that the Volume does disk +- Implement the Media record flag that indicates that the Volume does disk addressing. - Implement VolAddr, which is used when Volume is addressed like a disk, and form it from VolFile and VolBlock. @@ -92,7 +134,7 @@ For 1.33 resources, like Level? If so, I think I'd make it an optional directive in Job, Client, and Pool, with precedence such that Job overrides Client which in turn overrides Pool. -- Fix Ctl-C crashing the Console (readline?). +- Print a message when a job starts if the conf file is not current. - Finish work on conio.c - To pass Include 1 or two letter commands I Name Include name - first record @@ -118,6 +160,67 @@ For 1.33 F Number Number of filenames to follow ... + +- Spooling ideas taken from Volker Sauer's and other's emails: + > IMHO job spooling should be turned on + > + > 1) by job + > 2) by schedule + > 3) by sd + > + > where and 2) overrides 1) and 3) is independent. + + Yes, this is the minimum that I think is necessary. + + > + > Reason(s): + > It should be switched by job, because the job that backs up the machine + > with the bacula-sd on doesn't need spooling. + > It should be switched by schedule, because for full-backups I don't need + > spooling, so I can switch it off (because the network faster then the + > tapedrive) + + True, with the exception that if you have enough disk spool space, + and you want to run concurrent jobs, spooling can eliminate the block + interleaving restore inefficiencies. + + > And you should be able to turn it of by sd for sd-machines with low disk + > capacity or if you just don't need or want this feature. + > + > There should be: + > - definitly the possibility for multipe spool direcories + + Having multiple directories is no problem -- having different maximum + sizes creates specification problems. At some point, I will probably + have a common SD pool of spool directories as well as a set of + private spool directories for each device. The first implementation + will be a set of private spool directories for each device since + managing a global pool with a bunch of threads writing into the same + directory is *much* more complicated and prone to error. + + > - the ability to spool parts of a backup (not the whole client) + + This may change in the future, but for the moment, it will spool + either to a job high water mark, or until the directory is full + (reaches max spool size or I/O error). It will then write to tape, + truncate the spool file, and begin spooling again. + + > - spooling while writing to tape + + Not within a job, but yes, if you run concurrent jobs -- each is a + different thread. Within a job could be a feature, but *much* later. + + > - parallel spooling (like parallel jobs/ concurrent jobs) of clients + + Yes, this is one of my main motivations for doing it (aside from + eliminating tape "shoe shine" during incremental backups. + + > - flushing a backup that only went to disk (like amflush in amanda) + + This will be a future feature, since spooling is different from backing + up to disk. The future feature will be "migration" which will move a job + from one backup Volume to another. + - New Storage specifications: Passed to SD as a sort of BSR record called Storage Specification Record or SSR. @@ -146,6 +249,46 @@ For 1.33 In SD allow Device to have Multiple MediaTypes After 1.33: +- Look at www.nu2.nu/pebuilder as a helper for full windows + bare metal restore. +From Chris Hull: + it seems to be complaining about 12:00pm which should be a valid 12 + hour time. I changed the time to 11:59am and everything works fine. + Also 12:00am works fine. 0:00pm also works (which I don't think + should). None of the values 12:00pm - 12:59pm work for that matter. +Ideas from Jerry Scharf: + First let's point out some big pluses that bacula has for this + it's open source + more importantly it's active. Thank you so much for that + even more important, it's not flaky + it has an open access catalog, opening many possibilities + it's pushing toward heterogeneous systems capability + simple things: + I don't remember an include file directive for config files + (not filesets, actual config directives) + can you check the configs without starting the daemon? + some warnings about possible common mistakes + big things: + doing the testing and blessing of concurrent backup writes + this is absolutely necessary in the enterprise + easy user recovery GUI with full access checking + Macintosh file client + macs are an interesting niche, but I fear a server is a rathole + working bare iron recovery for windows + much better handling on running config changes + thinking through the logic of what happens to jobs in progress + the option for inc/diff backups not reset on fileset revision + a) use both change and inode update time against base time + b) do the full catalog check (expensive but accurate) + sizing guide (how much system is needed to back up N systems/files) + consultants on using bacula in building a disaster recovery system + an integration guide + or how to get at fancy things that one could do with bacula + logwatch code for bacula logs (or similar) + linux distro inclusion of bacula (brings good and bad, but necessary) + win2k/XP server capability (icky but you asked) + support for Oracle database ?? +=== - Look at adding SQL server and Exchange support for Windows. - Restore: Enter Filename: 'C:/Documents and Settings/Comercial/My Documents/MOP/formulário de registro BELAS ARTES.doc' causes Bacula to @@ -169,38 +312,10 @@ After 1.33: Job report (Volker Sauer). - Client does not show busy during Estimate command. - Implement Console mtx commands. -- Look at 2Gb limit for SQLite. -- Implement 3 Pools for a Job: - Job { -   Name = ... -   Full Backup Pool = xxx -   Incremental Backup Pool = yyy -   Differential Backup Pool = zzz - } - Add a default DB password to MySQL. GRANT all privileges ON bacula.* TO bacula@localhost IDENTIFIED BY 'bacula_password'; FLUSH PRIVILEGES; -- Define week of year for scheduler. W01, W02, ... - Week 01 of a year is per definition the first week that has the - Thursday in this year, which is equivalent to the week that contains the - fourth day of January. In other words, the first week of a new year is - the week that has the majority of its days in the new year. Week 01 - might also contain days from the previous year and the week before week - 01 of a year is the last week (52 or 53) of the previous year even if it - contains days from the new year. A week starts with Monday (day 1) and - ends with Sunday (day 7). For example, the first week of the year 1997 - lasts from 1996-12-30 to 1997-01-05 and can be written in standard - notation as - - 1997-W01 or 1997W01 - - The week notation can also be extended by a number indicating the day - of the week. For example, the day 1996-12-31, which is the Tuesday (day - 2) of the first week of 1997, can also be written as - - 1997-W01-2 or 1997W012 - - Implement a Mount Command and an Unmount Command where the users could specify a system command to be performed to do the mount, after which Bacula could attempt to @@ -254,7 +369,6 @@ After 1.33: - Take a careful look at Level for the estimate command, maybe make it a command line option. - Add Volume name to "I cannot write on this volume because" -- Make restore job check if all the files are actually restored. - Make tree walk routines like cd, ls, ... more user friendly by handling spaces better. - Write your PID file and chown root:wheel before drop. @@ -283,9 +397,7 @@ After 1.33: - Set IO_NOWAIT on Bacula TCP/IP packets. - Try doing a raw partition backup and restore by mounting a Windows partition. -- Report CVS problems to SourceForge. -- Implement .consolerc for Console -- From Lars Köllers: +- From Lars Kellers: Yes, it would allow to highly automatic the request for new tapes. If a tape is empty, bacula reads the barcodes (native or simulated), and if an unused tape is found, it runs the label command with all the @@ -294,8 +406,6 @@ After 1.33: By the way can bacula automatically "move" an empty/purged volume say in the "short" pool to the "long" pool if this pool runs out of volume space? -- Either restrict the characters in a name, or fix the problem - emailing with names containing / (smtp command line breaks). - Eliminate orphaned jobs: dbcheck, normal pruning, delete job command. Hm.  Well, there are the remaining orphaned job records: @@ -344,14 +454,9 @@ After 1.33: - Walk through the Pool records rather than the Job records in dird.c to create/update pools. - What to do about "list files job=xxx". -- Implement scan: for every slot it finds, zero the slot of - Volume other volume having that slot. - When job rescheduled, status gives is waiting for Client Rufus to connect to Storage File. Dir needs to inform SD that job is rescheduled. -- Fix get_storage_from_media_type (ua_restore) to use command line - storage= -- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume. - Make Dmsg look at global before calling subroutine. - Enable trace output at runtime for Win32 - Available volumes for autochangers (see patrick@baanboard.com 3 Sep 03 @@ -359,7 +464,6 @@ After 1.33: - Get and test MySQL 4.0 - Do a complete audit of all pthreads_mutex, cond, ... to ensure that any that are dynamically initialized are destroyed when no longer used. -- Write a mini-readline with history and editing. - Look at how fuser works and /proc/PID/fd that is how Nic found the file descriptor leak in Bacula. - Implement WrapCounters in Counters. @@ -370,24 +474,7 @@ After 1.33: - Can we dynamically change FileSets? - If pool specified to label command and Label Format is specified, automatically generate the Volume name. -- Take a careful look a the Basic recycling algorithm. When Bacula - chooses, the order should be: - - Look for Append - - Look for Recycle or Purged - - Prune volumes - - Look for purged - Instead of using lowest media Id, find the least recently used - volume. - - When the tape is mounted and Bacula requests the status - - Do everything possible to use it. - - Define a "available" status, which is the currently mounted - Volume and all volumes that are currently in the autochanger. - - Why can't SQL do the filename sort for restore? -- Is a pool specification really needed for a restore? Yes, and - you may want to exclude archive Pools. - Look at libkse (man kse) for FreeBSD threading. - Look into Microsoft Volume Shadowcopy Service VSS for backing up system state components (Active Directory, System Volume, ...) @@ -397,10 +484,7 @@ After 1.33: time as the user walks through the tree). - Possibly use the hash code if the user selects all for a restore command. - Orphaned Dir buffer at parse_conf.c:373 => store_dir -- Add Console usr permissions. - Fix "restore all" to bypass building the tree. -- Fix restore to list errors if Invalid block found, and if # files - restored does not match # expected. - Prohibit backing up archive device (findlib/find_one.c:128) - Implement Release Device in the Job resource to unmount a drive. - Implement Acquire Device in the Job resource to mount a drive, @@ -411,7 +495,6 @@ After 1.33: - Make things like list where a file is saved case independent for Windows. - Implement migrate -- Implement a PostgreSQL driver. - Bacula needs to propagate SD errors. > > cluster-dir: Start Backup JobId 252, Job=REUTERS.2003-08-11_15.04.12 > > prod4-sd: REUTERS.2003-08-11_15.04.12 Error: Write error on device @@ -432,9 +515,7 @@ After 1.33: We must cd into the directory then create the file without the full path name. - lstat() is not going to work on Win32 for testing date. -- Something is not right in last block of fill command. - Implement a Recycle command -- Add FileSet to command line arguments for restore. - Add client name to cram-md5 challenge so Director can immediately verify if it is the correct client. - Add JobLevel in FD status (but make sure it is defined). @@ -493,7 +574,6 @@ After 1.33: - Implement restore "current system", but take all files without doing selection tree -- so that jobs without File records can be restored. -- Implement a relocatable bacula.spec - Add prefixlinks to where or not where absolute links to FD. - Issue message to mount a new tape before the rewind. - Simplified client job initiation for portables. @@ -544,7 +624,6 @@ After 1.33: - Add UA rc and history files. - put termcap (used by console) in ./configure and allow -with-termcap-dir. -- Enhance time and size scanning routines. - Fix Autoprune for Volumes to respect need for full save. - Fix Win32 config file definition name on /install - Compare tape to Client files (attributes, or attributes and data) @@ -779,7 +858,7 @@ Need: ============================================================= - Request For Comments For File Backup Options + Request For Comments For File Backup Options 10 November 2002 Subject: File Backup Options @@ -985,6 +1064,7 @@ Done: (see kernsdone for more) - Implement ClientRunBeforeJob and ClientRunAfterJob. - Implement forward spacing block/file: position_device(bsr) -- just before read_block_from_device(); + === for 1.33 - Change console to bconsole. - Change smtp to bsmtp. @@ -1033,3 +1113,107 @@ Done: (see kernsdone for more) - Setup a standard job that builds a bootstrap file and saves it with the catalog database. - See if a restore job can add a file to the tape (prohibit this). +- Restrict characters permitted in a name. +- In restore, provide option for limiting to a particular Pool. +- In restore, list FileSets that only have different base names -- + i.e. any FileSet with the same name should be treated as the same. +- Make Scheduler sort jobs by StartTime, Priority. +- Make sure smtp and any other useful program is executable by the world + in case Bacula is not running as root. +- Look at Dan's field width problems in PostgreSQL. +- Look at effect of removing GROUP BYs. +- In restore take all filesets with same base name. +- From Alan Brown + BTW, there's a make install bug in 1.33 - with --enable-gnome, + gnome-console is built, but the binary and .conf are not being installed. +- Permit Bacula and apcupsd donations (not done for apcupsd). +- Fix Ctl-C crashing the Console (readline?). +- Look at code in recycle_oldes_purged_volume() recycle.c. Why not + let SQL do ORDER BY LastWritten ASC? +- Look at find_next_volume() algorithm. Currently, it selects: + +---------+------------+---------------------+-----------+ + | MediaId | VolumeName | LastWritten | VolBytes | + +---------+------------+---------------------+-----------+ + | 3 | Test13 | 0000-00-00 00:00:00 | 1 | + | 4 | Test14 | 0000-00-00 00:00:00 | 1 | + | 1 | test11 | 2003-12-03 18:39:55 | 4,004,926 | + | 2 | test12 | 2004-01-04 15:25:56 | 2,078,691 | + +---------+------------+---------------------+-----------+ + but perhaps it should fill already used Volumes first, and use + Append volumes before Purged, or Recycled, ... +- Possibly remove the "|| ap == NULL" on lines 123 and 207 of lib/var.c, + which creates compile problems on alpha systems. + var.c:123: no match for `va_list & == long int' +- Check "restore" 3 (JobId), then it asks for Storage resource. Does + it verify that the correct volume is chosen? +- Make Bacula "poll a drive". +- Notes for final checking of Nic's code: + Could I get you to double check the switch () statements in the + job_check_maxwaittime and job_check_maxruntime functions in + src/dird/job.c? +- Define week of year for scheduler. W01, W02, ... + Week 01 of a year is per definition the first week that has the + Thursday in this year, which is equivalent to the week that contains the + fourth day of January. In other words, the first week of a new year is + the week that has the majority of its days in the new year. Week 01 + might also contain days from the previous year and the week before week + 01 of a year is the last week (52 or 53) of the previous year even if it + contains days from the new year. A week starts with Monday (day 1) and + ends with Sunday (day 7). For example, the first week of the year 1997 + lasts from 1996-12-30 to 1997-01-05 and can be written in standard + notation as + 1997-W01 or 1997W01 + The week notation can also be extended by a number indicating the day + of the week. For example, the day 1996-12-31, which is the Tuesday (day + 2) of the first week of 1997, can also be written as + 1997-W01-2 or 1997W012 +- Either restrict the characters in a name, or fix the problem + emailing with names containing / (smtp command line breaks). +- Implement .consolerc for Console +- Implement scan: for every slot it finds, zero the slot of + Volume other volume having that slot. +- Make restore job check if all the files are actually restored. +- Look at 2Gb limit for SQLite. +- Fix get_storage_from_media_type (ua_restore) to use command line + storage= +- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume. +- Write a mini-readline with history and editing. +- Take a careful look a the Basic recycling algorithm. When Bacula + chooses, the order should be: + - Look for Append + - Look for Recycle or Purged + - Prune volumes + - Look for purged + Instead of using lowest media Id, find the least recently used + volume. + + When the tape is mounted and Bacula requests the status + - Do everything possible to use it. + + Define a "available" status, which is the currently mounted + Volume and all volumes that are currently in the autochanger. +- Is a pool specification really needed for a restore? Yes, and + you may want to exclude archive Pools. +- Implement a PostgreSQL driver. +- Fix restore to list errors if Invalid block found, and if # files + restored does not match # expected. +- Something is not right in last block of fill command. +- Add FileSet to command line arguments for restore. +- Enhance time and size scanning routines. +- Add Console usr permissions -- do by adding filters for + jobs, clients, storage, ... +- Put max network buffer size on a directive. +- Why does "mark cygwin" take so long!!!!!!!! +- Implement alist processing for ACLs from Console. +- When a file is set for restore, walk back up the chain of + directories, setting them to be restored. +- Figure out a way to set restore on a directory without recursively + decending. (recurse off?). +- Fix restore to only pull in last Differential and later Incrementals. +- Implement 3 Pools for a Job: + Job { +   Name = ... +   Full Backup Pool = xxx +   Incremental Backup Pool = yyy +   Differential Backup Pool = zzz + }