Update ChangeLog, ReleaseNotes, version, and catalog update scripts for version 5.2.0

[bacula/bacula] / bacula / ReleaseNotes

               Release Notes for Bacula 5.2.0

  Bacula code: Total files = 1,110 Total lines = 231,439 (Using SLOCCount)

General:
--------

This 5.2.0 version is a major release since the last version 5.0.3.
It includes well over 50,000 lines of changes made by 19 contributors
and a number of important new features:

Compatibility:
--------------
 As always, both the Director and Storage daemon must be upgraded at
 the same time.

 Older 5.0.x and 3.0.x File Daemons are compatible with the 5.2.0
 Director and Storage daemons. There should be no need to upgrade older File
 Daemons.

================== Warning !!!!!! ==========================

 The new pruning algorithm will not allow pruning of jobs that are essential to
 restore since the last Full backup. This is, of course, correct and desirable,
 but it is different from older Bacula versions that could inappropriately
 prune jobs. Pruning of volumes is not affected.

New Catalog format in version 5.2.0
-----------------------------------
This release of Bacula uses a new catalog format. We provide a set of
scripts that permit to convert a 5.0.x (version 12) catalog to 5.2.0 (version
14).  The Bacula project rpm spec files have been modified to
detect the database change and do the upgrade for you.  We strongly
recommend that you save a copy of your existing database before upgrading
using rpms.

The simplest way to upgrade the database assuming you are building from
source (i.e. manually):
1. Stop any current version of Bacula from running.
2. Save a copy of your existing database.
3. Configure, build, and install the 5.2.0 version of Bacula
   Note the new ./configure options in the manuals see the link
   at the bottom of this file.
4. cd <your-scripts-installation-directory>
5. ./update_bacula_tables
6. If you have multiple catalogs, edit the update_bacula_tables
   script and put the name of another database, and then
   execute the script again. Repeat this until all Bacula
   catalogs have been updated.
7. Start the new Bacula.  If everything worked, it should
   not print any error messages.

If you start from scratch, you don't need to run the update_bacula_tables
script because the create_bacula_tables script automatically creates the new
table format.  However, if you are using a version of Bacula older than 5.0.0
(e.g.  3.0.3) then you need to run the update_bacula_tables script that will
be found in the <bacula>/src/cats directory after you run the ./configure
command.  As mentioned above, before running this script, please backup your
catalog database, and when you do running it, be sure shutdown Bacula and be
aware that running the script can take some time depending on your database
size.

================================================================

Changes:

Speed and Memory Usage Enhancements:
- Implement LZO compression algorithm (need lzo2 headers to activate this option)
- New Pruning algoritm -- safer and faster
- Use of PostgreSQL cursor when doing large queries, should reduce the
  memory usage during restore and accurate backups.
- limit command in some list commands
- Bacula Virtual File System for fast catalog access
- Faster crc32 algorithm

Security Enhancements:
- Additional security against injection of illegal characters
- Fix possible fnmatch problem

Features Enhancements:
- Improvements in ACL and xattr handling on additional platforms:
  AIX, Darwind, FreeBSD, HPUX, IRIX, Solaris, Tru64
- Tray monitor for Windows
- New SQL backend and ./configure that permits building multiple catalog
  backends in one build, and has the main Bacula core code totally independent
  of the SQL backend, which should simplify packaging (only the
  libbacsql.so/dll needs to be changed to switch from one database backend
  to another (e.g. MySQL to PostgreSQL).
- Indent lsmark during restore
- Disable batch insert config item
- Director selection option in bconsole
- restorejob command in bconsole
- Many additional features for bat such as "brestore" panel
- Improved hash table for hard links
- Additional script variable editing such as %b (JobBytes), %F (JobFiles) 
  and %h (Client Address)

Other Enhancements:
- Includes better handling for Windows repares points, mount points, and 
  junction points.
- New database format
- Many new plugin API features
- Improved memory handling for restores
- Significant stability enhancements to bat
- xattr/ACL restore failure are now printed as Warning
- A few path name length limits in Windows have been removed.


===========================================================================

The 5.0.3 version is significant bug fix update to version 5.0.2.
It includes new code and some new features.
There is no database change since prior versions of 5.0.

Note, if you are running 5.0.0, read the 5.0.1 and 5.0.2 release notes below in
  the 5.0.1 release section.
  Important!!!

Compatibility:
--------------
 As always, both the Director and Storage daemon must be upgraded at
 the same time.

 Older 5.0.x and 3.0.x File Daemons are compatible with the 5.0.3
 Director and Storage daemons. There should be no need to upgrade older File
 Daemons.

Changes and New Features:
- If a VSS (Windows) snapshot fails, the whole Job is failed, previously
  it printed a warning message and continued.
- New editing codes %b = Job Bytes and %f = Job Files for runscripts
- Any non-Bacula project plugins (i.e. third party plugins) 
  must be upgraded to specify AGPLv3 in place of the GPLv2 used previously 
  for the PLUGIN_LICENSE.
- The bat Media page text columns can be sorted by clicking on the column.
- If possible, the SD will automatically update the catalog Volume size if 
  it does not match the Volume size. The mismatch between the Volume sizes
  typically occurs when the FD has a comm line drop or the FD crashes.


Bug fixes
1538 1554 1564 1567 1568 1569 1571 1577 1581 1582 1587 1594 1595 1606 1610

- Change license from GPLv2 to AGPLv3
- Make Win32 no filesystem change more explicit
- Backport Branch-4.0 release to Branch-5.0
- Massive backport from Branch-5.1 to Branch-5.0
- Fix postgresql catalog creation when version is not on the first line
- Fix bug #1610 handle empty xattr values on Linux and xBSD.
- Garbage collect memory pool at end of job and before sm_dump
- Add xattr seg fault protection suggested by Marco for bug #1610
- Add archlinux to os.m4
- More fixes for 32/64 bit problems in smartall
- Fix 32/64 bit problem in smartalloc dump routine
- Fix bad copy/paste in commit c88dccb88 prably a seg fault
- Add .dump and .exit commands for daemons
- Make SD automatically fix the Volume size in the Catalog when out of sync
- Use Pmsg in smartall.c rather than printf for tracing Windows
- Up maximum block size to 20M
- Fix for TLS bugs #1568 and #1599
- Improve comm line error handling for TLS, may fix bug #1568 and #1599
- Fix problem when sending Cancel event to plugin
- Convert all Jmsg on the watchdog thread to Qmsg
- Fix SD crash due to mismatched lock/unlock in error condition
- Fix bug #1587 if you have clients with different catalogs configured, both 
  catalogs contains all clients after a reload or restart of the dir
- Fix bug #1577 During migration: ERROR in block.c:950 Failed ASSERT: dev->is_open() 
  by moving and commenting out the ASSERT()
- Skip HB kill in FD if HB thread terminated
- Fix crash from unequal volume_lock/unlock calls
- New prunning algorithm -- should be more accurate
- Save any dumps during regression to dumps directory
- Fix crash from rw_lock/unlock miss match
- Attempt to fix duplicate job kill seg fault
- Replace pthread_kill by my_thread_send_signal() in jcr.c
- fix #1595 about batch mode detection problem for postgresql with non standard 
  install dir
- Fix #1594 about prune copy jobs
- Apply literal string fix in printf submitted by Luca Berra
- Fix bug #1582 Restore from multiple storage daemons breaks subsequent backups
- Fix segfault on "cancel jobid=" command
- Fix bug #1554 Windows installer doesn't honor /S flag
- Remove pool zap code that breaks 2drive-concurrent-test
- Fix #1571 and install libraries with 755 perms
- Fix bug #1581 ASSERTs in src/dird/jobq.c can never trigger
- Fix problem with BaseJob and Accurate FileSet options on client side
- Fix for bug #1569 deadlock/crash in Dir
- Fix segfault when loading Plugins
- Fix #1567 about display of long volume names truncated during restore
- Added new bacula-libs package.
- Fix bug #1538 Start Scripts return incorrect value
- Fix bug #1564 init scripts not LSB compliant

See ChangeLog for more details.

=============================================================================


============= Release 5.0.2 ========================

Changes since 5.0.1
-------------------

Bug fixes
1502 1511 1517 1524 1527 1532 1536 1541 1549 1551 1553 1559 1560

- Probable fix for SD crash bug #1553
- Fix #1559 problem when restoring pruned jobs with a regexp
- Fix for bug #1560 bcopy cannot find Volume
- Fix cancel crash bug #1551
- Check if sql backend is thread-safe
- Correct Pool display in SD status. Fixes bug #1541
- Fix cancel crash reported by Stephen Thompson
- Rewind on close to fix #1549
- Remove closelog() in bpipe fixes bug #1536
- Fix #1517 about missing Base level in .level command
- Replace ASSERT in block.c with fail Job
- Fix database locking calling db_lock and returning from function without
  calling db_unlock.
- Add missing db_unlock to bvfs_update_cache.
- Fix #1532 about permission on binaries
- Fix #1527 about deadlock during migration
- Another fix for OpenSSLv1
- Add -lrt to Solaris links
- Fix tls.c for OpenSSLv1
- Fix #1511 when trying to insert more than 50.000 directories in bvfs
- Fix plugin load not to stop if one plugin bad -- pointed out by James
- Remove --without-qwt from configure statement.
- Second correct fix to bug #1524 verify fails after adding or removing files
- Fix bug #1524 verify fails after adding or removing files
- Apply fix suggested by Andreas in bug #1502 for mediaview column sort problem
- Fix OpenSSL 1.x problem in crypto.c on Fedora 12
- Display AllowCompress warning message only if compression used in FileSet

=============================================================================


============= Release 5.0.1 ========================

!!!!!!!!!!!!!!! NOTE FOR 5.0.0 USERS !!!!!!!!!!!!!!!!!!!!
The Allow Duplicate Jobs directive has been significantly 
reworked, and the default value has changed. See below.

Truncate On Purge has been totally rewritten. See the new
features section of the manual.

When Volume Poll Interval is set in the SD DEVICE configuration,
(default 5 mins), after a certain number of polling tries (approx
10) polling will stop and the operator will be asked to
resolve the problem.  Previously there was no limit, and an
error message could be produced at each poll attempt.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Changes since 5.0.0
-------------------
- We believe that we have resolved most of the problems
  concerning canceled or failed jobs being "stuck" in the
  Director.  There is one outstanding problem in the SD when
  canceling jobs that we will fix in the next major release.
  If you see jobs that seem to be stuck, in general issuing
  a cancel command in bconsole should now make them go away.

Directives:
- The default for "Allow Duplicate Jobs" has been changed from 
  no to yes.  If you use this directive, please check your
  conf file, and note the next two items !!!!!!!!!!!!!!!!!!!
- AllowHigherDuplicates disabled. It did not work as documented   
    and was confusing.
- New directive "CancelLowerLevelDuplicates" See New Features
  section in the manual.
- Truncate on Purge rewritten.  See New Features section in the manual.

Bug fixes:
1448 1466 1467 1468 1476 1481 1486 1488 1494 1497 
1499 1501 1505 1509 1513

- Ensure SD asks for help when looping even if poll set. Fixes bug #1513.
- Fix three-pool regress bug
- Modify bacula.spec fixes bug #1505
- This version fixes an issue where the console window would start out
  docked. It is fixed by initiating the variables in the Pages class wi
  constructor.
- Fix make_catalog_backup.pl fails when catalog db is on other host
- Apply MacOSX installer patch from bug #1509
- Apply fix to previous fix of Copy problem. Fix proposed by reporter o
  #1476
- Fix bug #1501 -t does not print errors
- Apply SQLite3 update fix from bug #1497
- Apply bashism fix for diskchanger.in script from bug #1499
- Apply rpm fix for Sci Linux from bug #1494
- Take most recent Ukranian po from bug #1448
- Probable fix for Copy/Migration bug #1476
- Fix bug #1488 -- avoid recursion and race conditions in messages.c
- Upgrade cats library also to 5.0.0
- Fix missing console page in bat
- Add bat help files to Window install
- Improve Windows upgrade to ensure old FD is shutdown
- Fix bug #1481 -- bat consumes all console file descriptors
- Backport truncate on purge from 5.1.x
- Fix bug #1486 -- bat doesn't show any errors on command-line
- Update the bsock error URL
- Correct .my.cnf umask in make_catalog_backup.pl
- Apply fix for dbcheck use by make_catalog_backup.pl
- Fix seg fault in bscan from new comment field
- Allow multiple CNs when using TLS
- Fix seg fault in SQlite driver
- Make shared libs version the same as the Bacula release version
- Remove file_index sequential check
- Fix #1466 about Bogus pruning message

For Packagers:
1.  The default query.sql file is now, except for some comments, empty.
The old file, which we no longer support (it is impossible or difficult to
make it work on every backend, and the queries are mostly contributed) can
be found in <bacula-source>/examples/sample-query.sql.  The sample file is
not installed by the Makefiles

2.  When you install the mtx-changer script, you must also install
mtx-changer.conf if it does not exist.  This new file (mtx-changer.conf) is
required for mtx-changer to work, but it is a user configurable file, so on
any update, any existing file should not be overwritten.

3.  Bat should be built on every platform that is capabable of running Qt.
However, the Qt code is changing rather quickly and is not always
compatible from version to version.  We have built and verified bat on Qt
4.3.4.  We strongly recommend that you do not build and distribute bat with
any other version of Qt unless you personally test it.  To build against Qt
4.3.4, download the depkgs-qt package from the Bacula Source Forge download
location, read the README file and follow the instructions.

If you are building for Bacula version 5.0.0, please ensure that you do not
have qmake-qt4 loaded on your system.  If you do, either remove it or
rename it before trying to build bat.  If you do not, bat will probably be
built using the shared objects on your system.  For Bacula 5.0.1 and later,
this problem (bug) does not exist.

depkgs-qt does not install Qt on your system, nor does it interfere with
you having any other version of Qt installed on your system.  Once you
build bat with depkgs-qt, it should *not* use the Qt shared objects, but
rather they will be linked into the program.  After fully installing bat
(make install), you can run "ldd bat" to see what shared objects it will
use.  If any Qt shared objects are referenced, something has gone wrong.

4.  Unless absolutely necessary, we recommend that you do not define any
special library environment variables that apply to the ./configure -- for
example: LIBDIR=/...  ./configure <your-options> is strongly discouraged.
Doing so, could potentially cause Bacula to be linked against the wrong
shared objects.

5.  The Bacula project strongly recommends that you install Bacula into a
single directory, with a few minor exceptions such as the MySQL or
PostgreSQL databases.  Preferrably this should be /opt/bacula.  The full
recommendation is:

#!/bin/sh
# Recommended configure script for Bacula
prefix=/opt/bacula
email=xxx@yyy.zz
CFLAGS="-g -O2 -Wall" \
  ./configure \
    --sbindir=${prefix}/bin \
    --sysconfdir=${prefix}/etc \
    --docdir=${prefix}/html \
    --htmldir=${prefix}/html \
    --with-working-dir=${prefix}/working \
    --with-pid-dir=${prefix}/working \
    --with-subsys-dir=${prefix}/working \
    --with-scriptdir=${prefix}/scripts \
    --with-plugindir=${prefix}/plugins \
    --libdir=${prefix}/lib \
    --enable-smartalloc \
    --enable-tray-monitor \
    --enable-bat \
    --with-mysql \
    --with-dump-email=${email} \
    --with-job-email=${email} \
    --with-smtp-host=localhost \
    --with-baseport=9101

Obviously, the email, and some of the minor options (mysql, postgresql,
...) can be changed to suit your distribution, but the directory names
defined above are strongly recommended, and over time the default values in
the bacula-dir.conf and bacula-sd.conf will reflect these choices.

If you have any questions about this or would like a detailed document
describing our recommendations including packaging requirements, please
send an email to the bacula-devel list.

6.  Starting with Bacula version 3.0.0 up to Bacula 5.0.0, the shared
libraries that Bacula uses by default are named xxx-1.0.0.  Starting with
Bacula 5.0.1, we are going to name the libraries using the Bacula version.
So in Bacula 5.0.1, the libraries will be named xxx-5.0.1.  With future
versions, the last digit may or may not change when we distribute patch
updates (i.e.  the last digit of the version changes).  This will depend on
whether or not we have changed something in the library.  Hopefully this
new procedure will resolve some of the incompatibility problems between
different versions of the shared objects.

7.  The default build option for bconsole is conio (my own little console
routines).  I did this because some years ago, readline was very difficult
to maintain -- it and where it was found seemed to change on every release.
This generated at the time a number of support problems.  It seems to me
that since then there have been very few problems with readline.  As a
consequence, I have no problem if you want to make bconsole with readline
enabled.  It will actually give some very nice new bconsole command
completion functionality that Eric has written.  Bottom line: feel free to
use readline or not as you please.

==========================================================



Release Notes for Bacula 5.0.0

Release Numbering:
-----------------------------------------------------------
You might be wondering why this release jumps from 3.0.x to 
5.0.0 thus skipping version 4.0.0.  We have done this for 
several reasons: first, we wanted a way of distinguishing
the numbering system for the Bacula System Enterprise version
and the Bacula Project version.  So, to do so, we have decided
that the first number of the Bacula Project version will always
be odd, and the first number of the Enterprise version will always
be even. Thus the Bacula Project now moves from 3.0.x to 5.0.x.
In addition, we want to keep the Bacula Project version larger
than the Enterprise version to indicate that the Bacula Project
version is more advanced or has more features than the 
Enterprise version. For memory, the current Enterprise version
is 2.6.1, and the next release (in a few months -- before June 2010)
will be version 4.0.0.

Warning:
--------------------------------------------------------------
Please note the three Warnings below before doing any install or
upgrade.
       

Changes since 3.0.3a (the last Bacula Project release):
-------------------------------------------------------

Bug fixes:
1337 1351 1352 1354 1357 1361 1367 1369 1370 1373 
1376 1391 1403 1409 1420 1422 1427 1428 1431 1450 
1454 1455 1458 1456

Fixed:
 - Problem that locks the Director when the SMTP server wasn't 
   responding to the bsmtp program 
 - Restore's dir command shows incorrect file sizes 
 - Fix various problems with the reload command   
 - Fix problems with conio
 - Eliminate several deadlocks or potential race conditions from SD

Enhancements:
 - Project 5: Truncate volume after purge
 - Project 6: File Deduplication using Base Jobs 
 - Project 10: Restore from Multiple Storage Daemons
 - Project 11: AllowCompression per Device
 - Project 23: Add Maximum Concurent Jobs for Devices to balance load 
   between drives
 - Add Accurate Fileset Options to configure accurate detection. Can use 
   checksum verification for example.
 - Allow FD to keep root read but drop write permission (CAP)
 - Secure handling of passwords for catalog backup
 - Add Tab-completion for Bconsole when using Readline
 - Add Bvfs API to query the catalog without building a memory tree
 - Add new speed test to btape program
 - Add new Bat screens (Autochanger content, Job view, Media view, ...)
 - Windows version of Bat
 - Added Spanish Bacula translation   
 - New bconsole -u nnn option to have timeout on commands
 - Allow Migrate, Copy, and Virtual Full to read and write
   to the same Pool
 - show disabled   -- shows disabled Jobs
 - Add Pool File and Job retention periods (take precedence over Client)
 - Many ACL improvements
 - Level added to FD status Job entry
 - Allow turning on/off Block Checksum per device
 - Set Device Poll interval to 5 minutes -- previously did not poll
 - Implement lock manager with bad order protection

Compatibility:
As always, both the Director and Storage daemon must be upgraded at
the same time, and on any given machine, you must run only one version
of Bacula. This means that on the Director and Storage daemon machines,
you *must* upgrade your File daemon as well.

Older 3.0.x and possibly 2.4.x File Daemons are compatible with the 5.0.0
Director and Storage daemons. There should be no need to upgrade older File
Daemons immediately. However, we recommend that you do so as soon
as is reasonable for your situation.

================== Warning !!!!!! ==========================

New Catalog format in version 5.0.0 (3.1.9 or later)
-----------------------------------------------------

This release of Bacula uses a new catalog format. We provide a set of
scripts that permit to convert a 3.0.x (version 11) catalog to 5.0.0 (version
12).  The Bacula project rpms distributed on Source Forge will automatically
detect the database change and do the upgrade for you.  We strongly
recommend that you save a copy of your existing database before upgrading
using rpms.

The simplest way to upgrade the database assuming you are building from
source:
1. Stop any current version of Bacula from running.
2. Save a copy of your existing database.
3. Configure, build, and install the 5.0.0 version of Bacula
   Note the new ./configure options in the manuals see the link
   at the bottom of this file.
4. cd <your-scripts-installation-directory>
5. ./update_bacula_tables
6. If you have multiple catalogs, edit the update_bacula_tables
   script and put the name of another database, and then
   execute the script again. Repeat this until all Bacula
   catalogs have been updated.
7. Start the new Bacula.  If everything worked, it should
   not print any error messages.

If you start from scratch, you don't need to run the update_bacula_tables
script because the create_bacula_tables script automatically creates the new
table format.  However, if you are using a version of Bacula older than 3.1.9
(e.g.  3.0.3) then you need to run the update_bacula_tables script that will
be found in the <bacula>/src/cats directory after you run the ./configure
command.  As mentioned above, before running this script, please backup your
catalog database, and when you do running it, be sure shutdown Bacula and be
aware that running the script can take some time depending on your database
size.
================================================================


================== Warning !!!!!! ==========================
If you upgrade a previous Win32 client installation, you must
first stop the File daemon, then uninstall it before attempting
the upgrade. If you do not do so, the upgrade will probably
fail and you will need to manually find and run the 
Uninstall.exe file, before you can upgrade to the new version.
This is because the file locations are different.  If you want
to save your previous bacula-fd.conf file, please do so before
uninstalling it.
================================================================

================== Warning !!!!!! ==========================
Due to permissions problems on Windows Vista and later systems,
we no longer require SYSTEM or Administrator permissions to 
access the binaries and configuation files.  If you wish
to restrict access to these files, please see the New Features
section of the manual for how to set access control lists.
================================================================

Note, the Win32 Installer no longer installs the Bacula Servers
(Director and Storage daemon). See the New Features section of
the manual for more details.

============ Documentation improvements needed ==================

Note!!!!! The 5.0.0+ documentation has a number of known problems:

1. We have reduced the number of manuals, which has fixed many
   of the broken links found in the old 3.0.x version, but there
   are still some that need fixing.

2. Integration of the 3.0.x features into the main manuals has
   not yet been done.

The documentation will not be completed in the near future due to lack
manpower.  If you are unhappy with this situation as we are, we suggest
you help by submitting patches to the documentation.  Thanks.
=====


This release includes large number of new features and changes.
It should be more or less compatible with previous xx.conf files
and File daemons providing that you are not using any of the new
features.

Please see the New Features chapter of the new manual, which documents
some but not all the new features (we are still working on it):

http://www.bacula.org/5.0.x-manuals/en/main/main/New_Features_in_5_0_1.html
  
The ChangeLog has more details.