From e1c7670590b3654faa60842b60880987f7ad4516 Mon Sep 17 00:00:00 2001 From: Kern Sibbald Date: Wed, 25 Mar 2009 10:23:42 +0000 Subject: [PATCH] kes Added the following to provide solutions to the plugin/Accurate problem -- bug #1236 Cannot restore incremental backups with the Exchange plugin. - New Bacula read-only variable bVarAccurate -- returns accurate flag - New Bacula write-only variable bVarFileSeen -- marks a file as seen - New plugin entry point -- checkFile that is called at the end of an Accurate job and allows the plugin to mark a file as seen. git-svn-id: https://bacula.svn.sourceforge.net/svnroot/bacula/trunk@8595 91ce42f0-d328-0410-95d8-f526ca767f89 --- bacula/release-3-status.txt | 130 ++++++++++++++++++++++++++++++ bacula/src/console/console.c | 2 +- bacula/src/console/console_conf.c | 2 +- bacula/src/filed/accurate.c | 30 +++++-- bacula/src/filed/fd_plugins.c | 84 ++++++++++++++++++- bacula/src/filed/fd_plugins.h | 8 +- bacula/src/filed/protos.h | 3 +- bacula/src/filed/xattr.c | 2 +- bacula/src/lib/plugins.h | 2 +- bacula/technotes-2.5 | 57 +++---------- 10 files changed, 256 insertions(+), 64 deletions(-) create mode 100644 bacula/release-3-status.txt diff --git a/bacula/release-3-status.txt b/bacula/release-3-status.txt new file mode 100644 index 0000000000..5a12ad615e --- /dev/null +++ b/bacula/release-3-status.txt @@ -0,0 +1,130 @@ + Release 3.0 Status + +New features implemented: +- Accurate Backup +- Copy Jobs +- ACL code overhaul +- Solaris ZFS/NFSv4 ACL support +- Extended Attribute support +- Shared Objects +- Virtual Backup +- Catalog Format Enhancements + - Support for more than 4 Billion file entries + - New stats tables +- Duplicate Job Control +- TLS Authentication +- bextract non-portable Win32 data +- State File updated at Job Termination +- Bacula Plugins + - FD bpipe plugin + - FD Exchange plugin +- libdbi DB driver +- Console Enhancements + - status slots + - list joblog + - command separator +- Bootstrap regex +- Restore seek optimization +- Virtual Tape Emulation +- Bat Enhancements +- RunScript Enhancements +- Status enhancements +- FD connect timeout default to 3 minutes +- ftruncate implemented for NFS Volumes +- FD Version compatibily flag +- Statistics Enhancements +- Faster Attribute despooling +- Document split into 6 manuals +- GPLv2 license now compatible with OpenSSL +- Win64 File daemon +- Postmortum dump improvements in SD +- Deadlock mutex detection in SD + New Directives: +- Max Full Interval +- Max Diff Interval +- Honor No Dump Flag +- Exclude Dirs Containing +- Recycle Pool +- Max Run Sched Time +- Max Wait Time +- Full/Diff/Incr Max Run Time +- Scratch Pool +- Max Console Connections +- Spool Size (in Job resource) +- Allow Mixed Priorities +- Allow Duplicate Jobs +- Allow Higher Duplicates +- Cancel Queued Duplicates +- Cancel Running Duplicates +- TLS Authenticate +- Console (in RunScript) + +Project Items Completed for version 3.0.0: +Item 1: Accurate restoration of renamed/deleted files +Item 3: Merge multiple backups (Synthetic Backup or Consolidation) +Item 4: Implement Catalog directive for Pool resource in Director +Item 5: Add an item to the restore option where you can select a Pool +Item 8: Implement Copy pools +Item 12: Add Plug-ins to the FileSet Include statements. +Item 13: Restore only file attributes (permissions, ACL, owner, group...) +Item 18: Better control over Job execution +Item 26: Store and restore extended attributes, especially selinux file contexts +Item 27: make changing "spooldata=yes|no" possible for +Item 28: Implement an option to modify the last written date for volumes +Item n: Split documentation into several books + +New features to be documented: +- Update src/plugins/README +- command separator in console (!$%&'()*+,-/:;<>?[]^`{|}~) +- examples/database/dbcheck.sql +- dbport in dbcheck +- filepattern (restore with regex in bsr) +- Allow Mixed Priorities Kjetil Torgrim Homme 30Jun08 +- Eliminated license problems with OpenSSL. +- The exchange plugin truncates the exchange logs as soon as it has + sent the data to the sd. If the sd is spooling and something + goes wrong and the data doesn't make it to the backup medium then + the logs are lost, which could be a problem under some + restore scenarios. +- Regex filtered bsrs Kjetil Torgrim Homme 15Aug08 +- Added FD bVarAccurate and bVarFileSeen variables +- Added plugin entrypoint checkFile + + +Code to be completed before 3.0.0 is released: +2. Fix Exchange plugin bugs 1234,1235,1236,1237 + *1234 fixed. +4. Finish documentation (some noted above) +6. Make plugins work with Accurate -- needs two new API calls + - mark file not deleted + - is file deleted (at end of job) + - Pass accurate flag to Plugin +7. Proper installation of bat files +8. Turn off plugin options code until fully implemented. +9. Review whether or not plugins co-exist with encryption, + compression, ... + +Bat help files: +/usr/share/doc/bat/help (default) +Typical /usr/share/doc/xxx + changelog.Debian.gz + changelog.gz + CHANGES.txt.gz + copyright + CREDITS.txt + examples + FAQ.txt + help + README.Debian + README.txt + +Features that will not be completed: +- Plugin Options +- Dir plugin implementation somewhat minimal +- No example Dir plugin +- SD plugin implementation somewhat minimal +- No example SD plugin +- Install manual should be split into Install a + separate Configuration manual. +- Not all new features are tested or tested well + enough diff --git a/bacula/src/console/console.c b/bacula/src/console/console.c index c1f0d2c128..6098e92ad4 100644 --- a/bacula/src/console/console.c +++ b/bacula/src/console/console.c @@ -1,7 +1,7 @@ /* Bacula® - The Network Backup Solution - Copyright (C) 2000-2008 Free Software Foundation Europe e.V. + Copyright (C) 2000-2009 Free Software Foundation Europe e.V. The main author of Bacula is Kern Sibbald, with contributions from many others, a complete list can be found in the file AUTHORS. diff --git a/bacula/src/console/console_conf.c b/bacula/src/console/console_conf.c index ec40512c3f..d7c33a3098 100644 --- a/bacula/src/console/console_conf.c +++ b/bacula/src/console/console_conf.c @@ -1,7 +1,7 @@ /* Bacula® - The Network Backup Solution - Copyright (C) 2000-2007 Free Software Foundation Europe e.V. + Copyright (C) 2000-2009 Free Software Foundation Europe e.V. The main author of Bacula is Kern Sibbald, with contributions from many others, a complete list can be found in the file AUTHORS. diff --git a/bacula/src/filed/accurate.c b/bacula/src/filed/accurate.c index bb98acadd6..b343589854 100644 --- a/bacula/src/filed/accurate.c +++ b/bacula/src/filed/accurate.c @@ -1,7 +1,7 @@ /* Bacula® - The Network Backup Solution - Copyright (C) 2000-2008 Free Software Foundation Europe e.V. + Copyright (C) 2000-2009 Free Software Foundation Europe e.V. The main author of Bacula is Kern Sibbald, with contributions from many others, a complete list can be found in the file AUTHORS. @@ -43,6 +43,19 @@ typedef struct PrivateCurFile { bool seen; } CurFile; +bool accurate_mark_file_as_seen(JCR *jcr, char *fname) +{ + if (!jcr->accurate || !jcr->file_list) { + return false; + } + /* TODO: just use elt->seen = 1 */ + CurFile *temp = (CurFile *)jcr->file_list->lookup(fname); + if (temp) { + temp->seen = 1; /* records are in memory */ + } + return true; +} + static bool accurate_mark_file_as_seen(JCR *jcr, CurFile *elt) { /* TODO: just use elt->seen = 1 */ @@ -98,14 +111,15 @@ bool accurate_send_deleted_list(JCR *jcr) ff_pkt->type = FT_DELETED; foreach_htable(elt, jcr->file_list) { - if (!elt->seen) { /* already seen */ - Dmsg2(dbglvl, "deleted fname=%s seen=%i\n", elt->fname, elt->seen); - ff_pkt->fname = elt->fname; - ff_pkt->statp.st_mtime = elt->mtime; - ff_pkt->statp.st_ctime = elt->ctime; - encode_and_send_attributes(jcr, ff_pkt, stream); + if (elt->seen || plugin_check_file(jcr, elt->fname)) { + continue; } -// free(elt->fname); + Dmsg2(dbglvl, "deleted fname=%s seen=%i\n", elt->fname, elt->seen); + ff_pkt->fname = elt->fname; + ff_pkt->statp.st_mtime = elt->mtime; + ff_pkt->statp.st_ctime = elt->ctime; + encode_and_send_attributes(jcr, ff_pkt, stream); +// free(elt->fname); } term_find_files(ff_pkt); diff --git a/bacula/src/filed/fd_plugins.c b/bacula/src/filed/fd_plugins.c index 3b54d81cd4..6e547c6dca 100644 --- a/bacula/src/filed/fd_plugins.c +++ b/bacula/src/filed/fd_plugins.c @@ -150,6 +150,45 @@ void generate_plugin_event(JCR *jcr, bEventType eventType, void *value) return; } +/* + * Check if file was seen for accurate + */ +bool plugin_check_file(JCR *jcr, char *fname) +{ + Plugin *plugin; + bool ok = false; + int i = 0; + + if (!plugin_list || !jcr || !jcr->plugin_ctx_list) { + return false; /* Return if no plugins loaded */ + } + + bpContext *plugin_ctx_list = (bpContext *)jcr->plugin_ctx_list; + + Dmsg2(dbglvl, "plugin_ctx=%p JobId=%d\n", jcr->plugin_ctx_list, jcr->JobId); + + /* Pass event to every plugin */ + foreach_alist(plugin, plugin_list) { + jcr->plugin_ctx = &plugin_ctx_list[i++]; + jcr->plugin = plugin; + if (is_plugin_disabled(jcr)) { + continue; + } + if (plug_func(plugin)->checkFile == NULL) { + continue; + } + ok = plug_func(plugin)->checkFile(jcr->plugin_ctx, fname); + if (ok) { + break; + } + } + + jcr->plugin = NULL; + jcr->plugin_ctx = NULL; + return ok; +} + + /* * Sequence of calls for a backup: * 1. plugin_save() here is called with ff_pkt @@ -732,12 +771,12 @@ static boffset_t my_plugin_blseek(BFILE *bfd, boffset_t offset, int whence) static bRC baculaGetValue(bpContext *ctx, bVariable var, void *value) { JCR *jcr; - jcr = ((bacula_ctx *)ctx->bContext)->jcr; -// Dmsg1(dbglvl, "bacula: baculaGetValue var=%d\n", var); if (!value || !ctx) { return bRC_Error; } jcr = ((bacula_ctx *)ctx->bContext)->jcr; +// Dmsg1(dbglvl, "bacula: baculaGetValue var=%d\n", var); + jcr = ((bacula_ctx *)ctx->bContext)->jcr; if (!jcr) { return bRC_Error; } @@ -752,22 +791,61 @@ static bRC baculaGetValue(bpContext *ctx, bVariable var, void *value) Dmsg1(dbglvl, "Bacula: return my_name=%s\n", my_name); break; case bVarLevel: + *((int *)value) = jcr->get_JobLevel(); + Dmsg1(dbglvl, "Bacula: return bVarJobLevel=%d\n", jcr->get_JobLevel()); + break; case bVarType: + *((int *)value) = jcr->get_JobType(); + Dmsg1(dbglvl, "Bacula: return bVarJobType=%d\n", jcr->get_JobType()); + break; case bVarClient: + *((char **)value) = jcr->client_name; + Dmsg1(dbglvl, "Bacula: return Client_name=%s\n", jcr->client_name); + break; case bVarJobName: + *((char **)value) = jcr->Job; + Dmsg1(dbglvl, "Bacula: return Job name=%s\n", jcr->Job); + break; case bVarJobStatus: + *((int *)value) = jcr->JobStatus; + Dmsg1(dbglvl, "Bacula: return bVarJobStatus=%d\n", jcr->JobStatus); + break; case bVarSinceTime: + *((int *)value) = (int)jcr->mtime; + Dmsg1(dbglvl, "Bacula: return since=%d\n", (int)jcr->mtime); break; + case bVarAccurate: + *((int *)value) = (int)jcr->accurate; + Dmsg1(dbglvl, "Bacula: return accurate=%d\n", (int)jcr->accurate); + break; + case bVarFileSeen: + break; /* a write only variable, ignore read request */ } return bRC_OK; } static bRC baculaSetValue(bpContext *ctx, bVariable var, void *value) { + JCR *jcr; if (!value || !ctx) { return bRC_Error; } - Dmsg1(dbglvl, "bacula: baculaSetValue var=%d\n", var); + jcr = ((bacula_ctx *)ctx->bContext)->jcr; +// Dmsg1(dbglvl, "bacula: baculaGetValue var=%d\n", var); + jcr = ((bacula_ctx *)ctx->bContext)->jcr; + if (!jcr) { + return bRC_Error; + } +// Dmsg1(dbglvl, "Bacula: jcr=%p\n", jcr); + switch (var) { + case bVarFileSeen: + if (!accurate_mark_file_as_seen(jcr, (char *)value)) { + return bRC_Error; + } + break; + default: + break; + } return bRC_OK; } diff --git a/bacula/src/filed/fd_plugins.h b/bacula/src/filed/fd_plugins.h index 42dc47d34b..e5b0097e44 100644 --- a/bacula/src/filed/fd_plugins.h +++ b/bacula/src/filed/fd_plugins.h @@ -144,7 +144,9 @@ typedef enum { bVarClient = 5, bVarJobName = 6, bVarJobStatus = 7, - bVarSinceTime = 8 + bVarSinceTime = 8, + bVarAccurate = 9, + bVarFileSeen = 10 } bVariable; /* Events that are passed to plugin */ @@ -186,6 +188,7 @@ bool plugin_name_stream(JCR *jcr, char *name); int plugin_create_file(JCR *jcr, ATTR *attr, BFILE *bfd, int replace); bool plugin_set_attributes(JCR *jcr, ATTR *attr, BFILE *ofd); int plugin_save(JCR *jcr, FF_PKT *ff_pkt, bool top_level); +bool plugin_check_file(JCR *jcr, char *fname); #endif #ifdef __cplusplus @@ -227,7 +230,7 @@ typedef enum { #define FD_PLUGIN_MAGIC "*FDPluginData*" -#define FD_PLUGIN_INTERFACE_VERSION 2 +#define FD_PLUGIN_INTERFACE_VERSION 3 typedef struct s_pluginInfo { uint32_t size; @@ -259,6 +262,7 @@ typedef struct s_pluginFuncs { bRC (*pluginIO)(bpContext *ctx, struct io_pkt *io); bRC (*createFile)(bpContext *ctx, struct restore_pkt *rp); bRC (*setFileAttributes)(bpContext *ctx, struct restore_pkt *rp); + bool (*checkFile)(bpContext *ctx, char *fname); } pFuncs; #define plug_func(plugin) ((pFuncs *)(plugin->pfuncs)) diff --git a/bacula/src/filed/protos.h b/bacula/src/filed/protos.h index 428a9ddf7c..72300f18c6 100644 --- a/bacula/src/filed/protos.h +++ b/bacula/src/filed/protos.h @@ -1,7 +1,7 @@ /* Bacula® - The Network Backup Solution - Copyright (C) 2000-2008 Free Software Foundation Europe e.V. + Copyright (C) 2000-2009 Free Software Foundation Europe e.V. The main author of Bacula is Kern Sibbald, with contributions from many others, a complete list can be found in the file AUTHORS. @@ -53,6 +53,7 @@ bool parse_acl_stream(JCR *jcr, int stream); /* from accurate.c */ bool accurate_send_deleted_list(JCR *jcr); bool accurate_check_file(JCR *jcr, FF_PKT *ff_pkt); +bool accurate_mark_file_as_seen(JCR *jcr, char *fname); /* from backup.c */ bool encode_and_send_attributes(JCR *jcr, FF_PKT *ff_pkt, int &data_stream); diff --git a/bacula/src/filed/xattr.c b/bacula/src/filed/xattr.c index 2b01bff2d8..88e33a5a76 100644 --- a/bacula/src/filed/xattr.c +++ b/bacula/src/filed/xattr.c @@ -40,7 +40,7 @@ * * Written by Marco van Wieringen, November MMVIII * - * Version $Id: xattr.c 7879 2008-10-23 10:12:36Z kerns $ + * Version $Id$ */ #include "bacula.h" diff --git a/bacula/src/lib/plugins.h b/bacula/src/lib/plugins.h index 9934a4eb10..ead012e346 100644 --- a/bacula/src/lib/plugins.h +++ b/bacula/src/lib/plugins.h @@ -1,7 +1,7 @@ /* Bacula® - The Network Backup Solution - Copyright (C) 2007-2008 Free Software Foundation Europe e.V. + Copyright (C) 2007-2009 Free Software Foundation Europe e.V. The main author of Bacula is Kern Sibbald, with contributions from many others, a complete list can be found in the file AUTHORS. diff --git a/bacula/technotes-2.5 b/bacula/technotes-2.5 index c081cce26c..773fe455ef 100644 --- a/bacula/technotes-2.5 +++ b/bacula/technotes-2.5 @@ -1,53 +1,18 @@ Technical notes on version 2.5 -Projecs features implemented: -Items Completed for version 3.0.0: -Item 1: Accurate restoration of renamed/deleted files -Item 3: Merge multiple backups (Synthetic Backup or Consolidation) -Item 4: Implement Catalog directive for Pool resource in Director -Item 5: Add an item to the restore option where you can select a Pool -Item 8: Implement Copy pools -Item 12: Add Plug-ins to the FileSet Include statements. -Item 13: Restore only file attributes (permissions, ACL, owner, group...) -Item 18: Better control over Job execution -Item 26: Store and restore extended attributes, especially selinux file contexts -Item 27: make changing "spooldata=yes|no" possible for -Item 28: Implement an option to modify the last written date for volumes -Item n: Split documentation into several books - -New techical changes: -libtool -plugins -bpipe fd plugin -Exchange plugin - - -New features to be documented: -- Update src/plugins/README -- command separator in console (!$%&'()*+,-/:;<>?[]^`{|}~) -- examples/database/dbcheck.sql -- enhancement to wait command: wait mount ... -- dbport in dbcheck -- Allow Mixed Priorities Kjetil Torgrim Homme 30Jun08 -- Eliminated license problems with OpenSSL. -- The exchange plugin truncates the exchange logs as soon as it has - sent the data to the sd. If the sd is spooling and something - goes wrong and the data doesn't make it to the backup medium then - the logs are lost, which could be a problem under some - restore scenarios. -- Regex filtered bsrs Kjetil Torgrim Homme 15Aug08 - -Code to be completed before 3.0.0 is released: -*1. Fix bug #1221 problem with NODUMP flag -2. Fix Exchange plugin bugs 1234,1235,1236,1237 - *1234 fixed. -3. Resolve bug #1227 -4. Finish documentation (some noted above) -*5. Diff 2.4.4 to current SVN and check differences - - General: +25Mar09 +kes Added the following to provide solutions to the plugin/Accurate + problem -- bug #1236 Cannot restore incremental backups with + the Exchange plugin. + - New Bacula read-only variable bVarAccurate -- returns accurate flag + - New Bacula write-only variable bVarFileSeen -- marks a file as seen + - New plugin entry point -- checkFile that is called at the end + of an Accurate job and allows the plugin to mark a file as seen. +24Mar09 +kes Temporarly turn off comm timers because it causes bat to seg fault. + Must research making SIGUSR2 work with bat and Qt. 24Mar09 ebl Use MTIMEONLY fileset option in accurate check 23Mar09 -- 2.39.5