Update 2.4.1-b1 notes

[bacula/bacula] / bacula / ReleaseNotes

       Release Notes for Bacula 2.4.1-b1

  Bacula code: Total files = 500 Total lines = 188,654 (*.h *.c *.in)


==================   Warning !!!!!! ==========================
As of Version 2.4.0, the Win32 FD should not descend into 
any reparse point directory or reparse mount points unless the  
specified directory is explicitly mentioned at the top level of   
FileSet (as with Unix). A file that is linked to another file will
be backed up, much like Unix hardlinked files.  If you want 
reparse directories backed up, you must explicitly include them
in your FileSet.
==============================================================

This release consists of a number of bug fixes since version 2.4.0.
Important sections of the Storage daemon mount system have been
enhanced to correctly loading autochanger volumes when the drive
is empty and to handle manual tape changes.  Hopefully this will
resolve all the manual mount problems.

Bugs Fixed (hopefully -- not all confirmed):
 1103, 1100, 1105, 1094, 1109, 1106, 1101, 1102, 1112               

Release Beta version 2.4.1-b1
-  Back out bad src/lib/base64.c change.
-  Generally clean up the manual tape loading code. The main
     conceptial change is that when a volume is marked to be unloaded,
     its volume name is retained, and it is only marked as unloaded
     when either the autoloader says it is unloaded or another tape
     is read on that drive.
-  Fix a recycling problem with two autochangers reported in bug
     #1106. This may also (unlikely) fix a second recycling bug as
     reported in #1103.
-  Fix a SD lock volumes deadlock problem reported in bug #1100.
-  Fix format problem in bscan output reported in bug #1105.
-  Copy more data when restarting a job so that run
     overrides are kept. This should fix bug #1094.
-  Backport updates to jobq.c where possible.
-  Fix bug where SD did not ask operator if the device could not
     be opened. Reported by Eric.
-  Add dbuser to DIR conf file (replaces user).
-  Add --with-db-password to ./configure
-  Fix regress to handle db_password.
-  Fix a couple of pedantic compiler warnings.
-  Fix bug when starting a restore on a storage that doesn't
     have the requested MediaType.
-  More work on word alignments -- eliminate all ints from
     the configuration routines.
-  Fix bug reported by Scott Barninger where the bacula script
     refers to scripts in the wrong directory. Needed to meet the
     requirements of recent FHS location changes.
-  Fix word alignment problem on non-Intel 64 bit machines
     such as Solaris.  This should fix bug #1097 -- bus error in SD.
-  Add missing win32/patches/wxWidgets.patch to 2.4 repository.

=====================

Release 2.4.0
This Director and Storage daemon must be upgraded at the same time,
but they should be compatible with all 2.0.x File daemons, unless you
use some of the new features that affect the FD.  In other words, you should
not have to upgrade all your File daemons when you upgrade. There is
no database upgrade needed from version 2.0.x to 2.2.0 nor from version
2.2.x to version 2.4.0

This release consists mainly of a number of bug fixes since 2.2.x
but most importantly, a rewrite of significant portions of the 
Storage daemon reservation and autochanger code. This rewritten
code is simpler, more stable, and works much better, particularly
with multiple drive autochangers than the prior code did.  That 
said, there are still a number of issues to be resolved. Note,
the new code has not been tested in manual volume mount situations.

Bugs fixed:
 1095, 1086, 1091, 1070, 1041, 1083, 1078, 1066, 1068, 1038
 1062, 1065, 1046, 1047, 1031

New features:
- You can now build bat without the QWT libraries. Use
  --without-qwt as a ./configure option. 

Release Version 2.4.0
- Update version to 2.4.0 to reflect the magnitude of the SD changes.
- Make sure to clear JS_WaitMedia when operator mounts a Volume.
  This fixes bug #1095.
- Add create_postgresql_database.sql to Win32 installer. Should 
  fix problem reported by Juilio Monteiro.
- Get correct slot when auto unloading a device. This fixes bug
  #1086.
- Fix possible seg fault if SQL error.
- Fix renaming a Volume, which used the wrong Volume name. 
- If operator has rewind tape, print warning, release tape and
  try once more. If tape is positioned somewhere, something went
  wrong, so mark the tape in error and try once more. Previously
  this error was fatal, now it produces an error message.
- Ensure correct volume name displayed during restore
- Fix a few more Coverity reported problems.
- Fix #1091 about bad output in estimate command.
- Modify autochanger locking to attempt to avoid race
  conditions.
- Make sure device not busy before doing label command.
- Display open() errors except when polling. Previously too
  many were suppressed.  This should fix bug #1070.
- Fix Win32 reparse points. Bacula will not recurse into any
  reparse point directory, including mount points, unless the
  directory is explicitly mentioned at the top level (same as
  with Unix).  A file that is linked to another file will be
  backed up -- much as Unix does for hardlinked files.
  This fixes bug #1041.
- Remove StorageId test when pruning and recycling (Eric's changes).
- Fix buffer overruns detected by Coverity.
- Implement --without-qwt
- Fix layout of restore tree dialog
- Add configure bat QWT libraries, so that bat can be built
  with or without the QWT libaries.
- Implement regression that explicitly tests swapping a Volume
  from one drive to another.
- Enhance disk-changer to detect most error conditions.
- Fix SD code so that it properly swaps a Volume between drives.
  This fixes bug #1083.
- Prevent a Volume that is being swapped from being freed from
  the volume list. This will most likely fix, at least partially,
  bug #1083.
- Fix strippath so that it does not get a buffer overrun and crash FD.
  This fixes bug #1078.
- Make inability to change owner/group when creating a dir only a
  warning rather than an error.
- Fix SQL query in migration code
- Fix bat seg fault at termination.
- Add Bacula generated version to bat about box.
- Backport development stream SD reservation system changes.
- When wrong volume is mounted during read, unload_autochanger.
- Stop searching for Volumes in SD askdir if DIR returns the
  same volume name twice in a row.
- Rework class structures for VOLRES, DCR, and DEVICE to make
  the method names a bit more logical, and for more logically
  handling the responsibilities.
- Remove redundant code in terminating the scheduler that just
  causes a seg fault in many cases.
- Improve algorithm for detecting pre-reserved volume and
  swapping volumes.
- Prevent volume from being released while being swapped.
- Refactor parts of stored/mount.c
- Add sanity checks for VolWriteTime and VolReadTime
- Take care of bad clock changes while computing VolWriteTime
  and VolReadTime. This should fix or limit #1066
- Correct error string numbers in dird/catreq.c
- Restructure reserving, acquiring, and mounting volumes. Calls to
  autochanger are deferred for mount.c -- simplifes the code.
- Do not prune any running job. It just fails the job.
- Lock the volumes when changing dev->reserved_device and marking
  the volume unused otherwise the device can get reserved by
  another job before the volume is released, thus blocking it.
- Correct some SD catalog request error messages.
- Turn off code in read_record that causes a seg fault in the SD
  when reading past an EOS_LABEL.
- Turn off unloading the autochanger in reserve.c as it just
   causes problems.
- Lock volumes when unreserving a device.
- Do not mark volume unused when recycling.
- Fix bug #1068 fixes a SD crash when using Virtual autochanger.
- Generate correct JobMedia records during spooling/despooling when
  running concurrent jobs.  Thanks to Tom Ivar Helbekkmo
  <tih@hamartun.priv.no> for excellent analysis and testing.
- *Massive* commit of all fixes and back ports to create 2.2.9
  Mostly SD reservations enhancements.
- Fix Win32 FD backup/restore memory leak due to improper termination
  of BackupRead/Write. This fixes bug# 1038.
- Fixes a StorageId problem with volumes after migration.
- Implements autostart scripts for Debian 
- Fix mtx-changer.in for broken Debian mt program.
- Apply doc fix from bug #1062.
- Resolve crash and improper restore wx-console Win32 restore
  GUI. Fixes bug #1065.
- Update Win32 wxWidgets to latest version.
- Attempt to do correct handling of Win32 mount points. Should
  fix bug #1046.
- Add const to AIX prototype for initgroups()
- Fix to the JobMedia patch, which introduced a new problem.
- Fix creating first JobMedia record during Migration to include
  proper index. This caused slow restores of migrated jobs.
- Fix bug #1047, do not strip paths on symbolic links.
- Set catalog backup database and user name from values specified on 
  the ./configure line.
- Apply patch to correct bug #1031, about wrong pool source information
  in job report.


=========================================================================

Version 2.2.8 is a bug fix to version 2.2.7:
- It fixes bugs: 1036, 1033(doc), 1028, 1040, 1045(doc), 1030,
  1042(partially), and possibly bugs 1018 and 1032.
  Bugs 1018 and 1032 involve problems with multiple-drive autochangers,
  and are difficult to reproduce.   
  See the ChangeLog for more details.

Version 2.2.7 has several new features and several important bug fixes
  since version 2.2.6:
- It fixes bugs: 1009, 1022, 1024, 1019, 1012, 1021, 1020
  1007, 1008.  For more details, please see the technotes-2.1 file.
- Fixed a seg fault reported by Frank Sweetser that depended on
  exact path lengths, but the problem was in bsnprintf.c
- Double quoting include filenames (@xxx) in Bacula conf files is now
  possible, thanks to a patch by Michael Stapelberg.
- You can pipe input to include filenames (@|prog) by using a vertical
  bar, thanks to the above patch.
- A daylight savings time bug in the Win32 bsmpt was fixed by a patch from 
  Nerijus Baliunas.
- Marc Cousins submitted a patch that permits building the PostgreSQL
  driver with version 8.3.
- The configure option --archivedir has been changed to --with-archivedir

Version 2.2.6 is a minor bug fix release to version 2.2.5
- It fixes bugs: #1003, 942, 982, 990(response only), 991,   
  993, 986, 976.
- Fix bat crash wen it cannot connect to Director.
- Fix joblist failure bug in bat.

Version 2.2.5 is a major bug fix release to version 2.2.4 
- It fixes the following bugs: #961, 962, 963, 969, 968, 960,
  964, (possibly 935 and 903), 953, 953, 967, 966, 965, 954, 
  957, 908, 958, and 955.
- It also improves listing performance problems in bat pointed
  out by Chris Howells.

Version 2.2.4 is a minor bug fix release to version 2.2.3
- Possible fix for authorization problems bug #953.
- Possible fix for bug #908.
- Add waits to multiple exit detection code to try to force pid
  file to always be deleted.
- Restore good dev.tar.gz to rescue set appropriate binary property.
  This fixes bug #950.
- Fix seg fault in error exit of acquire_for_read after unsuccessfully
  trying to switch drives by checking for blocking before unblocking.
  Fixes bug #906.
- Cancel storage daemon in all cases where FD reports error. This
  should fix virtually all cases of bug #920 and will ensure that Devices
  are released as soon as possible.
- Fix error message that was clobbered when Dir tells SD it does not
  have write permission on Volume.  This should fix a minor point
  in bug #942, but not the main problem.
- Fix migration code to get correct Volume name with multiple volumes 
  by skipping |.  Fixes bug #936.
- Implement patch supplied by Landon to fix bug #944 where using
  TLS with bconsole uses 99+% of the CPU.
- Fix bug #946 about 'bacula-dir -t' which doesn't works
  as expected.
- Using 'm' in bconsole will show messages as in prior versions
  and not memory usage.

- Note, you need GTK >= 2.10 to be able to link the Tray Monitor
  program.

Version 2.2.3 is a critical bug fix release to version 2.2.2
- Fix bug #935, and possibly also bug #903 where files were not
  restored. MediaId was not properly set in JobMedia record after 
  a Volume change. This bug is known to affect all versions from 
  2.0.0 on, and probably versions prior to that.  A detailed analysis
  of the bug can be found at:
    
   http://www.bacula.org/downloads/bug-395.txt
   
- Add ./configure search in qwt-qt4 for qwt package
- Apply Martin Simmons patch that should turn off the new API usage
  when batch insert is turned off allowing building on older
  PostgreSQLs.
- Note, you need GTK >= 2.10 to be able to link the Tray Monitor 
  program.

Version 2.2.2 is a minor bug fix release to 2.2.1
- Detect if new PosgreSQL batch insert API is present.
- Correct incorrect mempool call causing Director crash on older
  systems that do not have va_copy().
- Update spec files for 2.2.1 release

Version 2.2.1 is mainly a minor bug fix release to version 2.2.0,
  the main changes since 2.2.0 are:
- Fixed bugs #921, 930, and 923.
- More portable timezone offset in bsmtp thaks to Attila Fülöp.
- Fix several bat issues.                     
- Use GTK+ calls instead of egg files in tray-monitor.
- Win32 service routines rewritten
- Updated projects file
Please see the ChangeLog for details.

Notes on 2.2.0 release follow:

Areas requiring caution or testing:
- You must have the thread safe version of MySQL client libraries loaded
  to build with MySQL enabled.
- Volumes are pruned only when absolutely necessary -- this may cause
  your database to grow compared to prior Bacula versions.
- Solaris door and even port files are no longer restored (equivalent
    to what we do with sockets).
- SQLite is no longer supported for Solaris -- it fails too often with
  bus errors.  SQLite3 should work on Solaris.
- The default setting for SQLite3 is now 'PRAGMA synchronous=OFF'.
  This makes it run 30 times faster, but increases the possiblity
  of a corrupted database if your server shuts down unexpectedly.
  The default behavior can be changed in src/version.h
- Restore on Win32, and in particular on Vista is untested. Please
  test before relying on it.  It should backup and restore reparse
  points.
- Win32 servers are untested, and very likely not to work.

The major new features are:
- Much faster insertion of attributes (somewhere around 10 times),
  many thanks to Eric Bollengier and Marc Cousin. 
- First release of bat (Bacula Administration Tool).  Note to build
  bat you must explicitly enable it on the ./configure line, you
  must have the Qt4 version 4.2 or later libraries loaded, the qwt
  (Qt Graphics) package loaded, and qmake and the other Qt4 tools 
  must be available. Most of the implementation was done by Dirk Bartley.
- Red/Black restore in memory tree (500 times faster loading).
- The Regex Where code to allow easier relocation of restored files 
  thanks to Eric.
- Socket level heartbeat for all connections (untested).
- posix_fadvise() use in the FD to improve performance reduces
  swapping due to opening/reading lots of files. Win32 equivalent
  implemented.
- Much improved Volume reservation code that should eliminate most
  conflicts experienced in multiple drive autochangers.
- Simpler locking in the SD in the reservation system.
- Detection of file size/date change during backup if enabled.
- New Recycle Pool feature -- thanks to Eric.
- Efficient implementation of very large include/exclude lists.
- Volumes are no longer pruned during 'status dir'
- Pruning is now more efficient, and if a Volume is purged,
  during pruning, it is immediately discovered.
- License is now GPL v2 without modifications, fix a few copyright
  mistakes made when adding FSFE copyright notice.

New ./configure options:
- --enable-bwx-console    
- --enable-bat
- --with-qwt=
- --with-db-name=
- --with-db-user=
- --enable-batch-insert

New bconsole commands:
- exec
- memory
- update recyclepool
- .sql
- .api
- .pwd (in restore tree)
- restoreclient (keyword)
- backupclient  (keyword)
- regexwhere    (keyword)
- update jobid  (new command)
- recyclepool   (keyword) 

New directives:
- Heartbeat Interval (bconsole.conf)
- Heartbeat Interval (bacula-dir.conf in Director, Client, Storage)
- TLS Allowed CN  (bacula-dir.conf in Client)
- Regex Where  (bacula-dir.conf in Job)
- Strip Prefix    (bacula-dir.conf in Job)
- Add Prefix   (bacula-dir.conf in Job)
- Add Suffex   (bacula-dir.conf in Job)
- Recycle Pool    (bacula-dir.conf in Pool)
- FailJobOnError  (bacula-dir.conf in RunScript)
- CheckFileChanges   (bacula-dir.conf in FileSet)
- StripPath    (bacula-dir.conf in FileSet)

Other features or bug fixes:
- Fixed bugs: 916, 910, 917, 914, 906, 907, 
  842, 830, 893, 861, 888, 886, 807, 877, 872
  885, 887, 864, 874, 882, 881, 863, 859, 
  856, 854, 845, 847, 846, 809, 772, 
  822, 825, 824, 808, 802, 797, 798, 795, 789, 791, 
  788, 780, 763, 782, 612, 767, 775, 621, 772, 764, 
  766, 621, 761, 755, 750, 741, 747, 742, 744, and 743.
- Configure bat with --enable-bat. Define qwt libraries with
  --with-qwt=<dir>. See Installation chapter of manual for details.
- Create a depkgs-qt package that has both Qt4 and qwt, which are
  needed to build bat. Most modern Linux systems will have both
  these packages in the distro.
- Storage overrides delete all previous storage definitions instead
  of prepending.
- One should be able to mount and unmount removable devices if the
  Device resource has Removeable Media set and the mount and unmount
  directives are defined.
- ./configure will do a better job of searching for qwt libraries.
- The Win32 version can no longer be shutdown from the tray monitor.
  Use the command line or the Services panne.
- Keep prune_volumes() from pruning the whole Scratch pool.
- More debug output in status storage.
- Correct moving a Scratch volume from pool to pool (some
  critical columns were lost).
- Different locking in reservations and despooling systems,
  which means more micro-locking and less macro-locking, which
  should give a lot more concurrency at the expense of slightly
  (<0.1%) more overhead due to more locking/unlocking, but  
  concurrent jobs should run much faster.
- Additional drive reservation algorithm that should solve a lot of
  the problems experienced with multiple drive autochangers.
- Storage daemon status command enhanced to more clearly show Volume,
  pool and media type when a job is waiting.
- Made bsmtp work with more strict SMTP servers.
- Detect doubly freed buffers in smartall.c
- wx-console renamed to bwx-console (Fedora request)
- gnome-console renamed to bgnome-console (Fedora request)
- Migration preserves original job's FileSetId
- Spooling implemented for migration jobs.
- Config files can be read through a pipe, by specifying a leading |
  in front of the configuration path/filename.
- New memory command (bconsole) that will print the current Director's
  memory usage, and an in use buffer dump.
- Console name changed from *Console* to -Console- to accomodate Win32
  filename restrictions.  
- Corrected the Win32 ftruncate bug.
- Additional version information added to Job reports.
- New -8 option for bsmtp to specify UTF-8 encoding type.
- bsmtp will no longer add < > to target (from, to, cc) if there
  is already a < in the target.
- Prefer Mounted Volumes extended to mean a volume that is being
  mounted by another job.
- Runscript timing set to same as version 1.38.11
- New TLS Allowed CN directive(s) permitted.
- New bconsole 'exec' command to run external script/command.
- Turn on FreeBSD/OpenBSD code to set EOT model on tape.
- Drop from root done before database is opened.
- Database user and name can be configured with:
  --with-db-name=name --with-db-user=user
- Turn on wait_for_drive in mtx-changer script.
- Rework how bar codes are handled in mtx-changer script.
  This appears to have been an unreported bug.
- Updated nagios plugin supplied by Christian Masopust
- Better restricted console ACL checking.
- New Client Connect Wait directive in Storage daemon so that
  users can configure how long SD waits for FD connection.
- Bacula will no longer permit pruning of the currently running
  job (note, if other jobs are running, they may be pruned).
- Sockets are no longer restored.
- dbcheck works in 300K chunks so will typically run faster.
- Include/Exclude lists can now be efficiently handled for quite
  large sizes -- tens of thousands of entries.
- Several memory leaks in migration and database usage were fixed.
- New console keyword restoreclient, which specifies the client to
  which the restore will be sent. The client keyword specifies 
  the backup client. The restoreclient keyword is optional if it is
  not specified, the backup client will also be the restore client.
- The disk file size is now checked to ensure that it agrees with the
  catalog value before Bacula will append to the disk (same as tape
  and DVD).