Release Notes for Bacula 5.2.0 Beta 1 Bacula code: Total files = 1,110 Total lines = 231,439 (Using SLOCCount) General: -------- This 5.2.0 version is a major release since the last version 5.0.3. It includes well over 50,000 lines of changes made by 19 contributors and a number of important new features: Compatibility: -------------- As always, both the Director and Storage daemon must be upgraded at the same time. Older 5.0.x and 3.0.x File Daemons are compatible with the 5.2.0 Director and Storage daemons. There should be no need to upgrade older File Daemons. ================== Warning !!!!!! ========================== The new pruning algorithm will not allow to prune jobs that are essential to restore since the last Full backup. Pruning of volumes are not affected. New Catalog format in version 5.2.0 ----------------------------------- This release of Bacula uses a new catalog format. We provide a set of scripts that permit to convert a 5.0.x (version 12) catalog to 5.2.0 (version 14). The Bacula project rpms distributed on Source Forge will automatically detect the database change and do the upgrade for you. We strongly recommend that you save a copy of your existing database before upgrading using rpms. The simplest way to upgrade the database assuming you are building from source: 1. Stop any current version of Bacula from running. 2. Save a copy of your existing database. 3. Configure, build, and install the 5.2.0 version of Bacula Note the new ./configure options in the manuals see the link at the bottom of this file. 4. cd 5. ./update_bacula_tables 6. If you have multiple catalogs, edit the update_bacula_tables script and put the name of another database, and then execute the script again. Repeat this until all Bacula catalogs have been updated. 7. Start the new Bacula. If everything worked, it should not print any error messages. If you start from scratch, you don't need to run the update_bacula_tables script because the create_bacula_tables script automatically creates the new table format. However, if you are using a version of Bacula older than 5.0.0 (e.g. 3.0.3) then you need to run the update_bacula_tables script that will be found in the /src/cats directory after you run the ./configure command. As mentioned above, before running this script, please backup your catalog database, and when you do running it, be sure shutdown Bacula and be aware that running the script can take some time depending on your database size. ================================================================ Changes: Speed and Memory Usage Enhancements: - Implement LZO compression algorithm (need lzo2 headers to activate this option) - New Pruning algoritm -- safer and faster - Use of PostgreSQL cursor when doing large queries, should reduce the memory usage during restore and accurate backups. - limit command in some list commands - Bacula Virtual File System for fast catalog access - Faster crc32 algorithm Security Enhancements: - Additional security against injection of illegal characters - Fix possible fnmatch problem Features Enhancements: - Improvements in ACL and xattr handling on additional platforms: AIX, Darwind, FreeBSD, HPUX, IRIX, Solaris, Tru64 - Tray monitor for Windows - New SQL backend and ./configure that permits building multiple catalog backends in one build, and has the main Bacula core code totally independent of the SQL backend, which should simplify packaging (only the libbacsql.so/dll needs to be changed. - Indent lsmark during restore - Disable batch insert config item - Director selection option in bconsole - restorejob command in bconsole - Many additional features for bat such as "brestore" panel - Improved hash table for hard links - Additional script variable editing such as %b (JobBytes), %F (JobFiles) and %h (Client Address) Other Enhancements: - Includes better handling for Windows repares points, mount points, and junction points. - New database format - Many new plugin API features - Improved memory handling for restores - Significant stability enhancements to bat - A few path name length limits in Windows have been removed. =========================================================================== The 5.0.3 version is significant bug fix update to version 5.0.2. It includes new code and some new features. There is no database change since prior versions of 5.0. Note, if you are running 5.0.0, read the 5.0.1 and 5.0.2 release notes below in the 5.0.1 release section. Important!!! Compatibility: -------------- As always, both the Director and Storage daemon must be upgraded at the same time. Older 5.0.x and 3.0.x File Daemons are compatible with the 5.0.3 Director and Storage daemons. There should be no need to upgrade older File Daemons. Changes and New Features: - If a VSS (Windows) snapshot fails, the whole Job is failed, previously it printed a warning message and continued. - New editing codes %b = Job Bytes and %f = Job Files for runscripts - Any non-Bacula project plugins (i.e. third party plugins) must be upgraded to specify AGPLv3 in place of the GPLv2 used previously for the PLUGIN_LICENSE. - The bat Media page text columns can be sorted by clicking on the column. - If possible, the SD will automatically update the catalog Volume size if it does not match the Volume size. The mismatch between the Volume sizes typically occurs when the FD has a comm line drop or the FD crashes. Bug fixes 1538 1554 1564 1567 1568 1569 1571 1577 1581 1582 1587 1594 1595 1606 1610 - Change license from GPLv2 to AGPLv3 - Make Win32 no filesystem change more explicit - Backport Branch-4.0 release to Branch-5.0 - Massive backport from Branch-5.1 to Branch-5.0 - Fix postgresql catalog creation when version is not on the first line - Fix bug #1610 handle empty xattr values on Linux and xBSD. - Garbage collect memory pool at end of job and before sm_dump - Add xattr seg fault protection suggested by Marco for bug #1610 - Add archlinux to os.m4 - More fixes for 32/64 bit problems in smartall - Fix 32/64 bit problem in smartalloc dump routine - Fix bad copy/paste in commit c88dccb88 prably a seg fault - Add .dump and .exit commands for daemons - Make SD automatically fix the Volume size in the Catalog when out of sync - Use Pmsg in smartall.c rather than printf for tracing Windows - Up maximum block size to 20M - Fix for TLS bugs #1568 and #1599 - Improve comm line error handling for TLS, may fix bug #1568 and #1599 - Fix problem when sending Cancel event to plugin - Convert all Jmsg on the watchdog thread to Qmsg - Fix SD crash due to mismatched lock/unlock in error condition - Fix bug #1587 if you have clients with different catalogs configured, both catalogs contains all clients after a reload or restart of the dir - Fix bug #1577 During migration: ERROR in block.c:950 Failed ASSERT: dev->is_open() by moving and commenting out the ASSERT() - Skip HB kill in FD if HB thread terminated - Fix crash from unequal volume_lock/unlock calls - New prunning algorithm -- should be more accurate - Save any dumps during regression to dumps directory - Fix crash from rw_lock/unlock miss match - Attempt to fix duplicate job kill seg fault - Replace pthread_kill by my_thread_send_signal() in jcr.c - fix #1595 about batch mode detection problem for postgresql with non standard install dir - Fix #1594 about prune copy jobs - Apply literal string fix in printf submitted by Luca Berra - Fix bug #1582 Restore from multiple storage daemons breaks subsequent backups - Fix segfault on "cancel jobid=" command - Fix bug #1554 Windows installer doesn't honor /S flag - Remove pool zap code that breaks 2drive-concurrent-test - Fix #1571 and install libraries with 755 perms - Fix bug #1581 ASSERTs in src/dird/jobq.c can never trigger - Fix problem with BaseJob and Accurate FileSet options on client side - Fix for bug #1569 deadlock/crash in Dir - Fix segfault when loading Plugins - Fix #1567 about display of long volume names truncated during restore - Added new bacula-libs package. - Fix bug #1538 Start Scripts return incorrect value - Fix bug #1564 init scripts not LSB compliant See ChangeLog for more details. ============================================================================= ============= Release 5.0.2 ======================== Changes since 5.0.1 ------------------- Bug fixes 1502 1511 1517 1524 1527 1532 1536 1541 1549 1551 1553 1559 1560 - Probable fix for SD crash bug #1553 - Fix #1559 problem when restoring pruned jobs with a regexp - Fix for bug #1560 bcopy cannot find Volume - Fix cancel crash bug #1551 - Check if sql backend is thread-safe - Correct Pool display in SD status. Fixes bug #1541 - Fix cancel crash reported by Stephen Thompson - Rewind on close to fix #1549 - Remove closelog() in bpipe fixes bug #1536 - Fix #1517 about missing Base level in .level command - Replace ASSERT in block.c with fail Job - Fix database locking calling db_lock and returning from function without calling db_unlock. - Add missing db_unlock to bvfs_update_cache. - Fix #1532 about permission on binaries - Fix #1527 about deadlock during migration - Another fix for OpenSSLv1 - Add -lrt to Solaris links - Fix tls.c for OpenSSLv1 - Fix #1511 when trying to insert more than 50.000 directories in bvfs - Fix plugin load not to stop if one plugin bad -- pointed out by James - Remove --without-qwt from configure statement. - Second correct fix to bug #1524 verify fails after adding or removing files - Fix bug #1524 verify fails after adding or removing files - Apply fix suggested by Andreas in bug #1502 for mediaview column sort problem - Fix OpenSSL 1.x problem in crypto.c on Fedora 12 - Display AllowCompress warning message only if compression used in FileSet ============================================================================= ============= Release 5.0.1 ======================== !!!!!!!!!!!!!!! NOTE FOR 5.0.0 USERS !!!!!!!!!!!!!!!!!!!! The Allow Duplicate Jobs directive has been significantly reworked, and the default value has changed. See below. Truncate On Purge has been totally rewritten. See the new features section of the manual. When Volume Poll Interval is set in the SD DEVICE configuration, (default 5 mins), after a certain number of polling tries (approx 10) polling will stop and the operator will be asked to resolve the problem. Previously there was no limit, and an error message could be produced at each poll attempt. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Changes since 5.0.0 ------------------- - We believe that we have resolved most of the problems concerning canceled or failed jobs being "stuck" in the Director. There is one outstanding problem in the SD when canceling jobs that we will fix in the next major release. If you see jobs that seem to be stuck, in general issuing a cancel command in bconsole should now make them go away. Directives: - The default for "Allow Duplicate Jobs" has been changed from no to yes. If you use this directive, please check your conf file, and note the next two items !!!!!!!!!!!!!!!!!!! - AllowHigherDuplicates disabled. It did not work as documented and was confusing. - New directive "CancelLowerLevelDuplicates" See New Features section in the manual. - Truncate on Purge rewritten. See New Features section in the manual. Bug fixes: 1448 1466 1467 1468 1476 1481 1486 1488 1494 1497 1499 1501 1505 1509 1513 - Ensure SD asks for help when looping even if poll set. Fixes bug #1513. - Fix three-pool regress bug - Modify bacula.spec fixes bug #1505 - This version fixes an issue where the console window would start out docked. It is fixed by initiating the variables in the Pages class wi constructor. - Fix make_catalog_backup.pl fails when catalog db is on other host - Apply MacOSX installer patch from bug #1509 - Apply fix to previous fix of Copy problem. Fix proposed by reporter o #1476 - Fix bug #1501 -t does not print errors - Apply SQLite3 update fix from bug #1497 - Apply bashism fix for diskchanger.in script from bug #1499 - Apply rpm fix for Sci Linux from bug #1494 - Take most recent Ukranian po from bug #1448 - Probable fix for Copy/Migration bug #1476 - Fix bug #1488 -- avoid recursion and race conditions in messages.c - Upgrade cats library also to 5.0.0 - Fix missing console page in bat - Add bat help files to Window install - Improve Windows upgrade to ensure old FD is shutdown - Fix bug #1481 -- bat consumes all console file descriptors - Backport truncate on purge from 5.1.x - Fix bug #1486 -- bat doesn't show any errors on command-line - Update the bsock error URL - Correct .my.cnf umask in make_catalog_backup.pl - Apply fix for dbcheck use by make_catalog_backup.pl - Fix seg fault in bscan from new comment field - Allow multiple CNs when using TLS - Fix seg fault in SQlite driver - Make shared libs version the same as the Bacula release version - Remove file_index sequential check - Fix #1466 about Bogus pruning message For Packagers: 1. The default query.sql file is now, except for some comments, empty. The old file, which we no longer support (it is impossible or difficult to make it work on every backend, and the queries are mostly contributed) can be found in /examples/sample-query.sql. The sample file is not installed by the Makefiles 2. When you install the mtx-changer script, you must also install mtx-changer.conf if it does not exist. This new file (mtx-changer.conf) is required for mtx-changer to work, but it is a user configurable file, so on any update, any existing file should not be overwritten. 3. Bat should be built on every platform that is capabable of running Qt. However, the Qt code is changing rather quickly and is not always compatible from version to version. We have built and verified bat on Qt 4.3.4. We strongly recommend that you do not build and distribute bat with any other version of Qt unless you personally test it. To build against Qt 4.3.4, download the depkgs-qt package from the Bacula Source Forge download location, read the README file and follow the instructions. If you are building for Bacula version 5.0.0, please ensure that you do not have qmake-qt4 loaded on your system. If you do, either remove it or rename it before trying to build bat. If you do not, bat will probably be built using the shared objects on your system. For Bacula 5.0.1 and later, this problem (bug) does not exist. depkgs-qt does not install Qt on your system, nor does it interfere with you having any other version of Qt installed on your system. Once you build bat with depkgs-qt, it should *not* use the Qt shared objects, but rather they will be linked into the program. After fully installing bat (make install), you can run "ldd bat" to see what shared objects it will use. If any Qt shared objects are referenced, something has gone wrong. 4. Unless absolutely necessary, we recommend that you do not define any special library environment variables that apply to the ./configure -- for example: LIBDIR=/... ./configure is strongly discouraged. Doing so, could potentially cause Bacula to be linked against the wrong shared objects. 5. The Bacula project strongly recommends that you install Bacula into a single directory, with a few minor exceptions such as the MySQL or PostgreSQL databases. Preferrably this should be /opt/bacula. The full recommendation is: #!/bin/sh # Recommended configure script for Bacula prefix=/opt/bacula email=xxx@yyy.zz CFLAGS="-g -O2 -Wall" \ ./configure \ --sbindir=${prefix}/bin \ --sysconfdir=${prefix}/etc \ --docdir=${prefix}/html \ --htmldir=${prefix}/html \ --with-working-dir=${prefix}/working \ --with-pid-dir=${prefix}/working \ --with-subsys-dir=${prefix}/working \ --with-scriptdir=${prefix}/scripts \ --with-plugindir=${prefix}/plugins \ --libdir=${prefix}/lib \ --enable-smartalloc \ --enable-tray-monitor \ --enable-bat \ --with-mysql \ --with-dump-email=${email} \ --with-job-email=${email} \ --with-smtp-host=localhost \ --with-baseport=9101 Obviously, the email, and some of the minor options (mysql, postgresql, ...) can be changed to suit your distribution, but the directory names defined above are strongly recommended, and over time the default values in the bacula-dir.conf and bacula-sd.conf will reflect these choices. If you have any questions about this or would like a detailed document describing our recommendations including packaging requirements, please send an email to the bacula-devel list. 6. Starting with Bacula version 3.0.0 up to Bacula 5.0.0, the shared libraries that Bacula uses by default are named xxx-1.0.0. Starting with Bacula 5.0.1, we are going to name the libraries using the Bacula version. So in Bacula 5.0.1, the libraries will be named xxx-5.0.1. With future versions, the last digit may or may not change when we distribute patch updates (i.e. the last digit of the version changes). This will depend on whether or not we have changed something in the library. Hopefully this new procedure will resolve some of the incompatibility problems between different versions of the shared objects. 7. The default build option for bconsole is conio (my own little console routines). I did this because some years ago, readline was very difficult to maintain -- it and where it was found seemed to change on every release. This generated at the time a number of support problems. It seems to me that since then there have been very few problems with readline. As a consequence, I have no problem if you want to make bconsole with readline enabled. It will actually give some very nice new bconsole command completion functionality that Eric has written. Bottom line: feel free to use readline or not as you please. ========================================================== Release Notes for Bacula 5.0.0 Release Numbering: ----------------------------------------------------------- You might be wondering why this release jumps from 3.0.x to 5.0.0 thus skipping version 4.0.0. We have done this for several reasons: first, we wanted a way of distinguishing the numbering system for the Bacula System Enterprise version and the Bacula Project version. So, to do so, we have decided that the first number of the Bacula Project version will always be odd, and the first number of the Enterprise version will always be even. Thus the Bacula Project now moves from 3.0.x to 5.0.x. In addition, we want to keep the Bacula Project version larger than the Enterprise version to indicate that the Bacula Project version is more advanced or has more features than the Enterprise version. For memory, the current Enterprise version is 2.6.1, and the next release (in a few months -- before June 2010) will be version 4.0.0. Warning: -------------------------------------------------------------- Please note the three Warnings below before doing any install or upgrade. Changes since 3.0.3a (the last Bacula Project release): ------------------------------------------------------- Bug fixes: 1337 1351 1352 1354 1357 1361 1367 1369 1370 1373 1376 1391 1403 1409 1420 1422 1427 1428 1431 1450 1454 1455 1458 1456 Fixed: - Problem that locks the Director when the SMTP server wasn't responding to the bsmtp program - Restore's dir command shows incorrect file sizes - Fix various problems with the reload command - Fix problems with conio - Eliminate several deadlocks or potential race conditions from SD Enhancements: - Project 5: Truncate volume after purge - Project 6: File Deduplication using Base Jobs - Project 10: Restore from Multiple Storage Daemons - Project 11: AllowCompression per Device - Project 23: Add Maximum Concurent Jobs for Devices to balance load between drives - Add Accurate Fileset Options to configure accurate detection. Can use checksum verification for example. - Allow FD to keep root read but drop write permission (CAP) - Secure handling of passwords for catalog backup - Add Tab-completion for Bconsole when using Readline - Add Bvfs API to query the catalog without building a memory tree - Add new speed test to btape program - Add new Bat screens (Autochanger content, Job view, Media view, ...) - Windows version of Bat - Added Spanish Bacula translation - New bconsole -u nnn option to have timeout on commands - Allow Migrate, Copy, and Virtual Full to read and write to the same Pool - show disabled -- shows disabled Jobs - Add Pool File and Job retention periods (take precedence over Client) - Many ACL improvements - Level added to FD status Job entry - Allow turning on/off Block Checksum per device - Set Device Poll interval to 5 minutes -- previously did not poll - Implement lock manager with bad order protection Compatibility: As always, both the Director and Storage daemon must be upgraded at the same time, and on any given machine, you must run only one version of Bacula. This means that on the Director and Storage daemon machines, you *must* upgrade your File daemon as well. Older 3.0.x and possibly 2.4.x File Daemons are compatible with the 5.0.0 Director and Storage daemons. There should be no need to upgrade older File Daemons immediately. However, we recommend that you do so as soon as is reasonable for your situation. ================== Warning !!!!!! ========================== New Catalog format in version 5.0.0 (3.1.9 or later) ----------------------------------------------------- This release of Bacula uses a new catalog format. We provide a set of scripts that permit to convert a 3.0.x (version 11) catalog to 5.0.0 (version 12). The Bacula project rpms distributed on Source Forge will automatically detect the database change and do the upgrade for you. We strongly recommend that you save a copy of your existing database before upgrading using rpms. The simplest way to upgrade the database assuming you are building from source: 1. Stop any current version of Bacula from running. 2. Save a copy of your existing database. 3. Configure, build, and install the 5.0.0 version of Bacula Note the new ./configure options in the manuals see the link at the bottom of this file. 4. cd 5. ./update_bacula_tables 6. If you have multiple catalogs, edit the update_bacula_tables script and put the name of another database, and then execute the script again. Repeat this until all Bacula catalogs have been updated. 7. Start the new Bacula. If everything worked, it should not print any error messages. If you start from scratch, you don't need to run the update_bacula_tables script because the create_bacula_tables script automatically creates the new table format. However, if you are using a version of Bacula older than 3.1.9 (e.g. 3.0.3) then you need to run the update_bacula_tables script that will be found in the /src/cats directory after you run the ./configure command. As mentioned above, before running this script, please backup your catalog database, and when you do running it, be sure shutdown Bacula and be aware that running the script can take some time depending on your database size. ================================================================ ================== Warning !!!!!! ========================== If you upgrade a previous Win32 client installation, you must first stop the File daemon, then uninstall it before attempting the upgrade. If you do not do so, the upgrade will probably fail and you will need to manually find and run the Uninstall.exe file, before you can upgrade to the new version. This is because the file locations are different. If you want to save your previous bacula-fd.conf file, please do so before uninstalling it. ================================================================ ================== Warning !!!!!! ========================== Due to permissions problems on Windows Vista and later systems, we no longer require SYSTEM or Administrator permissions to access the binaries and configuation files. If you wish to restrict access to these files, please see the New Features section of the manual for how to set access control lists. ================================================================ Note, the Win32 Installer no longer installs the Bacula Servers (Director and Storage daemon). See the New Features section of the manual for more details. ============ Documentation improvements needed ================== Note!!!!! The 5.0.0+ documentation has a number of known problems: 1. We have reduced the number of manuals, which has fixed many of the broken links found in the old 3.0.x version, but there are still some that need fixing. 2. Integration of the 3.0.x features into the main manuals has not yet been done. The documentation will not be completed in the near future due to lack manpower. If you are unhappy with this situation as we are, we suggest you help by submitting patches to the documentation. Thanks. ===== This release includes large number of new features and changes. It should be more or less compatible with previous xx.conf files and File daemons providing that you are not using any of the new features. Please see the New Features chapter of the new manual, which documents some but not all the new features (we are still working on it): http://www.bacula.org/5.0.x-manuals/en/main/main/New_Features_in_5_0_1.html The ChangeLog has more details.