From: Kern Sibbald Date: Fri, 10 May 2002 15:08:16 +0000 (+0000) Subject: Mostly documentation of pruning X-Git-Tag: Release-1.20~12 X-Git-Url: https://git.sur5r.net/?a=commitdiff_plain;h=92c2ecbb5fcaab8a6b8b5c9352de5701ec7e378e;p=bacula%2Fbacula Mostly documentation of pruning git-svn-id: https://bacula.svn.sourceforge.net/svnroot/bacula/trunk@17 91ce42f0-d328-0410-95d8-f526ca767f89 --- diff --git a/bacula/ChangeLog b/bacula/ChangeLog index 09d01728ac..4123a82625 100644 --- a/bacula/ChangeLog +++ b/bacula/ChangeLog @@ -1,3 +1,137 @@ +2002-05-10 Release 1.19 +- Allow the user to select a new period for pruning. +- Lots of additions to the manual -- prune and purge + commands documented. +- Applied Phil's configure.in fix for --prefix, ... +- Fixed bug found by Phil (patch supplied) in updating + MD5 signatures (revert to 32 bit FileId, move "static" + variables into JCR) (catreq.c fd-cmds.c). +- Reverted to using INTEGER for FileId in make_sql_tables + due to a bug in MySQL. +- Change editing code to %d for FileId. +- Remove sqlite in make distclean in cats directory. +- Remove console.conf in console during make distclean. +- Remove gnome-console.conf during make distclean. +- Remove bacula-dir.conf during make distclean. +- Set default level when using Console if none specified. + Bug reported by Phil. +- A simple . command from Console is ignored. +- Change program named from filed to bacual-fd in winmain.cpp +- Change default config file for Win32 in winmain.cpp +- Free namebuf on early return from find_one.c. Bug reported by Phil. +- Modify testfind.c to dump orphaned buffers. +- Removed terabytes from parse_conf.c because of problems with + older gcc compilers. +- Turn off gnome options in gnome-console by constructing empty argv. +- Fixes to Verify when only MD5 differs. +- Insert 0 for MD5 as default rather than space. +- Allow .messages "transparent" command while reading input + in UserAgent server. +- In dird/verify.c ensure that correct filename is printed if only + the MD5 differs. Minor reindenting caused large diff. +- Delete unused code from backup.c +- In filed/verify.c ensure that same algorithm as backup.c is used + to pass back MD5 signatures -- especially for directories and files + that cannot be read. Change dummy filename from X to *MD5-id*. + This dummy value should never be printed. +- Make gnome-console poll Director every 5 seconds for output. + This means that queued up messages are displayed at reasonable + intervals. Delete some unused code hanging around from the tty + console program. +- Begin implementation of prune commands. +- Add command line history to gnome-console. Not yet saved across sessions. +- Fixed some broken URLs in the manual. +- Added JobId type. +- Wrote first cut of "prune files" and "prune jobs". +- Added command line history to gnome-console 2500 lines max. +- Widened store_time() and associated variables to 64 bits using + new btime_t definition. +- Removed GNOME about box and replaced it with + a somewhat crude Gtk+ about box in the gnome-console. +- Widened StartDay in the Job DB record to be 64 bits. +- Changed edit_uint_ to edit_uint64_ everywhere. Much more + descriptive of what is done. +- Removed most llds and replaced them with %s and edit_uint64. + This makes the code a bit easier to read for beginners. +- Added a btime_t typedef which is 64 bits wide. +- Added most of the code needed to do Purge and Prune of + database. Not yet tested. Please do not use. +- Additional config options for Console. +- Started adding code for Automatic Recycling of Volumes. +- Allow arbitrary length filenames in Verify code. +- Fix incorrect filename in Verify code. +- Significant enhancement to number scanning in config parser. +- Requires initializing MySQL tables, or applying cats/alter.sql + to modify Media table. +- Modified Makefile to printer a much more visible + message if the make ends in error. +- Added the Purge value to the VolStatus in the Media table. + This value is set if the Volume is purged. +- Enhanced the db_delete_media_record() routine to delete + all associated records in the database. +- Modified Console to always write to the variable "output" + rather than stdout. This will permit directing output to a file. +- Enhanced the Console configuration file to have a Console + resource. This will allow us to add an rcfile and history file. +- Modified Find_next_vol in catreq.c to search for "Recycle" + volumes if there are none marked Append. Also made Get_Vol_info + return the VolStatus to the Storage daemon. +- Allow upper/lower case match on job level names. +- Added another warning message to the Console "delete media" command. +- Corrected a number of FileId types from uint64_t to FileId_t in + ua_retention.c. +- In ua server close database before freeing JCR (because pointer is + kept in ua and jcr structures). +- parse all numbers as doubles. Do lots of checking for valid number + formats. Allow numbers in scientific form (e.g. 1.5e+10). +- Remove terabyte modifier as it doesn't work in some compilers. +- Fix bug in handling some modifiers. +- Added all necessary code for Recycle to Storage daemon. A Volume + marked Recycle will now be overwritten. +- Filled out the prune command a bit and did some testing. +- Make console accept redirected input. +- Altered the Table definitions to include Recycle, + FileRetention, JobRetention, and AutoPrune. +- Widened StartDay to 64 bits. +- Use JobId_t in more places. +- Added the new table fields to the database record definitions. +- Changed Recycle from string to a binary quantity. +- Added a Version table with a VersionId to detect. + future changes in the database. This should prevent + a Bacula from working with a database that is not in sync. +- Modify Console to accept input from a file. This will permit + the .read command and allow reading a .rc file. +- Added new retention and recycle variables to the Director's configuration. +- The UA subroutines or commands can now be called from core + code because the output routines detect the absence of a + UA socket and direct output to the Job. +- Added a verbose flage to the ua packet to permit reduction of + output while running a UA command (prune) from a Job. +- Did a fair amount of work on the prune command. Prune Volume now works. +- Purge Volume now works. +- Made last changes for integrating prune and purge commands. +- Add -ltermcap to CONS_LIBS when readline is configured. + More work to be done to search for termcap. +- Added cats/drop_sqlite_tables.in, which will delete the SQLite database. +- Got CWEB working so that we can compile filesys.w and immortal.w +- Modified depkgs to include cweb. +- Note, CWEB is not yet used by the core Bacula code. +- Made cats/alter.sql, which alters an old database to bring it + up to the new format. This only works with MySQL since SQLite does + not have the ALTER SQL command. +- Changed the old StartDay field in the db to be JobTDate, which is + the latest time/date in widened Unix time format that the Job ran. + This value is used when doing pruning. +- Added code in cats/sql.c to verify that the database internal version + corresponds to the db version compiled in Bacula. It is set to 1 currently. +- Lots of changes to cats to bring the SQL up to date with the new + retention period changes. +- Added Console command code to permit changing a Volume's retention period. +- Removed old code that used date_encode() and replaced it with widened + Unix time(). +- Started modifying Message resource scanner so that we can have multiple + message resources. Much more work to be done. +- Moved scanning for time into new library routine string_to_btime(). 2002-04-22 Release 1.18 - Applied Phil's configure.in fix for --prefix, ... diff --git a/bacula/src/cats/bdb_update.c b/bacula/src/cats/bdb_update.c index e0eba75249..d75187dcda 100755 --- a/bacula/src/cats/bdb_update.c +++ b/bacula/src/cats/bdb_update.c @@ -9,6 +9,8 @@ * system. * * Kern Sibbald, January MMI + * + * $Id: */ /* @@ -139,7 +141,7 @@ int db_update_media_record(B_DB *mdb, MEDIA_DBR *mr) mr->PoolId = omr.PoolId; mr->VolMaxBytes = omr.VolMaxBytes; mr->VolCapacityBytes = omr.VolCapacityBytes; - strcpy(mr->Recycle, omr.Recycle); + mr->Recycle = omr.Recycle; fseek(mdb->mediafd, omr.rec_addr, SEEK_SET); if (fwrite(mr, len, 1, mdb->mediafd) != 1) { diff --git a/bacula/src/dird/bacula-dir.conf.in b/bacula/src/dird/bacula-dir.conf.in index ee60d0d15b..f6bae84b51 100644 --- a/bacula/src/dird/bacula-dir.conf.in +++ b/bacula/src/dird/bacula-dir.conf.in @@ -65,6 +65,9 @@ Client { FDPort = @fd_port@ Catalog = MyCatalog Password = "@fd_password@" # password for FileDaemon + File Retention = 6m # six months + Job Retention = 1y # one year + AutoPrune = yes # Prune expired Jobs/Files } @@ -130,4 +133,7 @@ Messages { Pool { Name = Default Pool Type = Backup + Recycle = yes # Bacula can automatically recycle Volumes + AutoPrune = yes # Prune expired volumes + Volume Retention = 1y # one year } diff --git a/bacula/src/dird/dird_conf.c b/bacula/src/dird/dird_conf.c index 6218f539ab..7d327814e2 100644 --- a/bacula/src/dird/dird_conf.c +++ b/bacula/src/dird/dird_conf.c @@ -18,6 +18,8 @@ * for the resource records. * * Kern Sibbald, January MM + * + * $Id: */ /* Copyright (C) 2000, 2001, 2002 Kern Sibbald and John Walker @@ -108,9 +110,9 @@ static struct res_items cli_items[] = { {"fdport", store_pint, ITEM(res_client.FDport), 0, ITEM_REQUIRED, 0}, {"password", store_password, ITEM(res_client.password), 0, ITEM_REQUIRED, 0}, {"catalog", store_res, ITEM(res_client.catalog), R_CATALOG, 0, 0}, - {"fileretention", store_time, ITEM(res_client.FileRetention), 0, ITEM_DEFAULT, 60*60*24*30}, - {"jobretention", store_time, ITEM(res_client.JobRetention), 0, ITEM_DEFAULT, 60*60*24*365}, - {"autoprune", store_yesno, ITEM(res_client.AutoPrune), 1, ITEM_DEFAULT, 0}, + {"fileretention", store_time, ITEM(res_client.FileRetention), 0, ITEM_DEFAULT, 60*60*24*60}, + {"jobretention", store_time, ITEM(res_client.JobRetention), 0, ITEM_DEFAULT, 60*60*24*180}, + {"autoprune", store_yesno, ITEM(res_client.AutoPrune), 1, ITEM_DEFAULT, 1}, {NULL, NULL, NULL, 0, 0, 0} }; @@ -216,8 +218,8 @@ static struct res_items pool_items[] = { {"acceptanyvolume", store_yesno, ITEM(res_pool.accept_any_volume), 1, 0, 0}, {"catalogfiles", store_yesno, ITEM(res_pool.catalog_files), 1, ITEM_DEFAULT, 1}, {"volumeretention", store_time, ITEM(res_pool.VolRetention), 0, ITEM_DEFAULT, 60*60*24*365}, - {"autoprune", store_yesno, ITEM(res_pool.AutoPrune), 1, ITEM_DEFAULT, 0}, - {"recycle", store_yesno, ITEM(res_pool.Recycle), 1, ITEM_DEFAULT, 0}, + {"autoprune", store_yesno, ITEM(res_pool.AutoPrune), 1, ITEM_DEFAULT, 1}, + {"recycle", store_yesno, ITEM(res_pool.Recycle), 1, ITEM_DEFAULT, 1}, {NULL, NULL, NULL, 0, 0, 0} }; diff --git a/bacula/src/dird/protos.h b/bacula/src/dird/protos.h index 0c693d8b9a..8cd2a22ba5 100644 --- a/bacula/src/dird/protos.h +++ b/bacula/src/dird/protos.h @@ -1,5 +1,7 @@ /* * Director external function prototypes + * + * $Id: */ /* Copyright (C) 2000, 2001, 2002 Kern Sibbald and John Walker @@ -35,13 +37,13 @@ extern char *level_to_str(int level); /* fd_cmds.c */ extern int connect_to_file_daemon(JCR *jcr, int retry_interval, - int max_retry_time, int verbose); + int max_retry_time, int verbose); extern int send_include_list(JCR *jcr); extern int send_exclude_list(JCR *jcr); extern int get_attributes_and_put_in_catalog(JCR *jcr); extern int get_attributes_and_compare_to_catalog(JCR *jcr, int last_full_id); extern int put_file_into_catalog(JCR *jcr, long file_index, char *fname, - char *link, char *attr, int stream); + char *link, char *attr, int stream); /* job.c */ extern void free_jcr(JCR *jcr); @@ -53,7 +55,7 @@ extern void mount_request(JCR *jcr, BSOCK *bs, char *buf); /* msgchan.c */ extern int connect_to_storage_daemon(JCR *jcr, int retry_interval, - int max_retry_time, int verbose); + int max_retry_time, int verbose); extern int start_storage_daemon_job(JCR *jcr); extern int start_storage_daemon_message_thread(JCR *jcr); extern int32_t bget_msg(BSOCK *bs, int type); @@ -62,5 +64,5 @@ extern int response(BSOCK *fd, char *resp, char *cmd); /* newvol.c */ extern int newVolume(JCR *jcr); -/* ua_cmds.c */ +/* ua_cmd.c */ extern int create_pool(B_DB *db, POOL *pool); diff --git a/bacula/src/dird/ua.h b/bacula/src/dird/ua.h index 3fa6f25a3b..2336a4b2b2 100644 --- a/bacula/src/dird/ua.h +++ b/bacula/src/dird/ua.h @@ -2,6 +2,8 @@ * Includes specific to the Director User Agent Server * * Kern Sibbald, August MMI + * + * $Id: */ /* Copyright (C) 2000, 2001, 2002 Kern Sibbald and John Walker @@ -31,19 +33,19 @@ typedef struct s_ua_context { JCR *jcr; B_DB *db; CAT *catalog; - char *cmd; /* return command/name buffer */ - char *args; /* command line arguments */ - char *argk[MAX_ARGS]; /* argument keywords */ - char *argv[MAX_ARGS]; /* argument values */ - int argc; /* number of arguments */ - char **prompt; /* list of prompts */ - int max_prompts; /* max size of list */ - int num_prompts; /* current number in list */ - int auto_display_messages; /* if set, display messages */ + char *cmd; /* return command/name buffer */ + char *args; /* command line arguments */ + char *argk[MAX_ARGS]; /* argument keywords */ + char *argv[MAX_ARGS]; /* argument values */ + int argc; /* number of arguments */ + char **prompt; /* list of prompts */ + int max_prompts; /* max size of list */ + int num_prompts; /* current number in list */ + int auto_display_messages; /* if set, display messages */ int user_notified_msg_pending; /* set when user notified */ - int automount; /* if set, mount after label */ - int quit; /* if set, quit */ - int verbose; /* set for normal UA verbosity */ + int automount; /* if set, mount after label */ + int quit; /* if set, quit */ + int verbose; /* set for normal UA verbosity */ } UAContext; /* ua_cmds.c */ @@ -75,7 +77,7 @@ int select_pool_and_media_dbr(UAContext *ua, POOL_DBR *pr, MEDIA_DBR *mr); void start_prompt(UAContext *ua, char *msg); void add_prompt(UAContext *ua, char *prompt); int do_prompt(UAContext *ua, char *msg, char *prompt); -CAT *get_catalog_resource(UAContext *ua); +CAT *get_catalog_resource(UAContext *ua); STORE *get_storage_resource(UAContext *ua, char *cmd); int get_media_type(UAContext *ua, char *MediaType); int get_pool_dbr(UAContext *ua, POOL_DBR *pr); @@ -85,3 +87,4 @@ int get_job_dbr(UAContext *ua, JOB_DBR *jr); int find_arg_keyword(UAContext *ua, char **list); int do_keyword_prompt(UAContext *ua, char *msg, char **list); +int confirm_retention(UAContext *ua, btime_t *ret, char *msg); diff --git a/bacula/src/dird/ua_prune.c b/bacula/src/dird/ua_prune.c index b9d9e49241..d790e78c82 100644 --- a/bacula/src/dird/ua_prune.c +++ b/bacula/src/dird/ua_prune.c @@ -4,6 +4,8 @@ * Applies retention periods * * Kern Sibbald, February MMII + * + * $Id: */ /* @@ -202,20 +204,29 @@ int prunecmd(UAContext *ua, char *cmd) N_("Volume"), NULL}; if (!open_db(ua)) { - return 1; + return 01; } switch (find_arg_keyword(ua, keywords)) { case 0: client = select_client_resource(ua); + if (!client || !confirm_retention(ua, &client->FileRetention, "File")) { + return 0; + } prune_files(ua, client); return 1; case 1: client = select_client_resource(ua); + if (!client || !confirm_retention(ua, &client->JobRetention, "Job")) { + return 0; + } prune_jobs(ua, client); return 1; case 2: if (!select_pool_and_media_dbr(ua, &pr, &mr)) { - return 1; + return 0; + } + if (!confirm_retention(ua, &mr.VolRetention, "Volume")) { + return 0; } prune_volume(ua, &pr, &mr); return 1; @@ -225,21 +236,24 @@ int prunecmd(UAContext *ua, char *cmd) switch (do_keyword_prompt(ua, _("Choose item to prune"), keywords)) { case 0: client = select_client_resource(ua); - if (!client) { - return 1; + if (!client || !confirm_retention(ua, &client->FileRetention, "File")) { + return 0; } prune_files(ua, client); break; case 1: client = select_client_resource(ua); - if (!client) { - return 1; + if (!client || !confirm_retention(ua, &client->JobRetention, "Job")) { + return 0; } prune_jobs(ua, client); break; case 2: if (!select_pool_and_media_dbr(ua, &pr, &mr)) { - return 1; + return 0; + } + if (!confirm_retention(ua, &mr.VolRetention, "Volume")) { + return 0; } prune_volume(ua, &pr, &mr); return 1; @@ -279,14 +293,18 @@ int prune_files(UAContext *ua, CLIENT *client) Dmsg1(050, "select sql=%s\n", query); if (!db_sql_query(ua->db, query, file_count_handler, (void *)&del)) { - bsendmsg(ua, "%s", db_strerror(ua->db)); + if (ua->verbose) { + bsendmsg(ua, "%s", db_strerror(ua->db)); + } Dmsg0(050, "Count failed\n"); goto bail_out; } if (del.tot_ids == 0) { - bsendmsg(ua, _("No Files found for client %s to prune from %s catalog.\n"), - client->hdr.name, client->catalog->hdr.name); + if (ua->verbose) { + bsendmsg(ua, _("No Files found for client %s to prune from %s catalog.\n"), + client->hdr.name, client->catalog->hdr.name); + } goto bail_out; } @@ -315,7 +333,7 @@ int prune_files(UAContext *ua, CLIENT *client) db_sql_query(ua->db, query, NULL, (void *)NULL); Dmsg1(050, "Del sql=%s\n", query); } - bsendmsg(ua, _("%d Files for client %s pruned from %s catalog.\n"), del.num_ids, + bsendmsg(ua, _("Pruned %d Files for client %s from %s catalog.\n"), del.num_ids, client->hdr.name, client->catalog->hdr.name); bail_out: @@ -395,7 +413,9 @@ int prune_jobs(UAContext *ua, CLIENT *client) Mmsg(&query, insert_delcand, ed1, cr.ClientId); if (!db_sql_query(ua->db, query, NULL, (void *)NULL)) { - bsendmsg(ua, "%s", db_strerror(ua->db)); + if (ua->verbose) { + bsendmsg(ua, "%s", db_strerror(ua->db)); + } Dmsg0(050, "insert delcand failed\n"); goto bail_out; } @@ -405,14 +425,18 @@ int prune_jobs(UAContext *ua, CLIENT *client) Dmsg1(100, "select sql=%s\n", query); if (!db_sql_query(ua->db, query, count_handler, (void *)&cnt)) { - bsendmsg(ua, "%s", db_strerror(ua->db)); + if (ua->verbose) { + bsendmsg(ua, "%s", db_strerror(ua->db)); + } Dmsg0(050, "Count failed\n"); goto bail_out; } if (cnt.count == 0) { - bsendmsg(ua, _("No Jobs for client %s found to prune from %s catalog.\n"), - client->hdr.name, client->catalog->hdr.name); + if (ua->verbose) { + bsendmsg(ua, _("No Jobs for client %s found to prune from %s catalog.\n"), + client->hdr.name, client->catalog->hdr.name); + } goto bail_out; } @@ -448,7 +472,7 @@ int prune_jobs(UAContext *ua, CLIENT *client) db_sql_query(ua->db, query, NULL, (void *)NULL); Dmsg1(050, "Del sql=%s\n", query); } - bsendmsg(ua, _("%d Jobs for client %s pruned from %s catalog.\n"), del.num_ids, + bsendmsg(ua, _("Pruned %d Jobs for client %s from %s catalog.\n"), del.num_ids, client->hdr.name, client->catalog->hdr.name); bail_out: @@ -486,8 +510,10 @@ int prune_volume(UAContext *ua, POOL_DBR *pr, MEDIA_DBR *mr) } if (cnt.count == 0) { - bsendmsg(ua, "There are no Jobs associated with Volume %s. It is purged.\n", - mr->VolumeName); + if (ua->verbose) { + bsendmsg(ua, "There are no Jobs associated with Volume %s. It is purged.\n", + mr->VolumeName); + } if (!mark_media_purged(ua, mr)) { goto bail_out; } @@ -504,7 +530,9 @@ int prune_volume(UAContext *ua, POOL_DBR *pr, MEDIA_DBR *mr) Mmsg(&query, "SELECT JobId FROM JobMedia WHERE MediaId=%d", mr->MediaId); if (!db_sql_query(ua->db, query, file_delete_handler, (void *)&del)) { - bsendmsg(ua, "%s", db_strerror(ua->db)); + if (ua->verbose) { + bsendmsg(ua, "%s", db_strerror(ua->db)); + } Dmsg0(050, "Count failed\n"); goto bail_out; } @@ -534,7 +562,7 @@ int prune_volume(UAContext *ua, POOL_DBR *pr, MEDIA_DBR *mr) if (del.JobId) { free(del.JobId); } - bsendmsg(ua, _("%d Jobs on Volume %s pruned from catalog.\n"), del.num_del, + bsendmsg(ua, _("Pruned %d Jobs on Volume %s from catalog.\n"), del.num_del, mr->VolumeName); /* If purged, mark it so */ @@ -553,7 +581,9 @@ static int mark_media_purged(UAContext *ua, MEDIA_DBR *mr) strcmp(mr->VolStatus, "Full") == 0) { strcpy(mr->VolStatus, "Purged"); if (!db_update_media_record(ua->db, mr)) { - bsendmsg(ua, "%s", db_strerror(ua->db)); + if (ua->verbose) { + bsendmsg(ua, "%s", db_strerror(ua->db)); + } return 0; } } diff --git a/bacula/src/dird/ua_purge.c b/bacula/src/dird/ua_purge.c index 834d129058..552d8d9096 100644 --- a/bacula/src/dird/ua_purge.c +++ b/bacula/src/dird/ua_purge.c @@ -7,6 +7,8 @@ * Purges Jobs from Volumes * * Kern Sibbald, February MMII + * + * $Id: */ /* diff --git a/bacula/src/dird/ua_select.c b/bacula/src/dird/ua_select.c index 60d8a3739a..f0329268a6 100644 --- a/bacula/src/dird/ua_select.c +++ b/bacula/src/dird/ua_select.c @@ -3,6 +3,8 @@ * Bacula Director -- User Agent Prompt and Selection code * * Kern Sibbald, October MMI + * + * $Id: */ /* @@ -41,6 +43,36 @@ void start_prompt(UAContext *ua, char *msg); STORE *select_storage_resource(UAContext *ua); JOB *select_job_resource(UAContext *ua); +/* + * Confirm a retention period + */ +int confirm_retention(UAContext *ua, btime_t *ret, char *msg) +{ + char ed1[30]; + + for ( ;; ) { + bsendmsg(ua, _("The current %s retention period is: %s\n"), + msg, edit_btime(*ret, ed1)); + if (!get_cmd(ua, _("Continue? (yes/mod/no): "))) { + return 0; + } + if (strcasecmp(ua->cmd, _("mod")) == 0) { + if (!get_cmd(ua, _("Enter new retention period: "))) { + return 0; + } + if (!string_to_btime(ua->cmd, ret)) { + bsendmsg(ua, _("Invalid period.\n")); + continue; + } + continue; + } + if (strcasecmp(ua->cmd, _("yes")) == 0) { + break; + } + } + return 1; +} + /* * Given a list of keywords, find the first one * that is in the argument list. diff --git a/bacula/src/version.h b/bacula/src/version.h index 01f063f51c..ee81553dff 100644 --- a/bacula/src/version.h +++ b/bacula/src/version.h @@ -1,8 +1,8 @@ /* */ #define VERSION "1.19" #define VSTRING "1" -#define DATE "09 May 2002" -#define LSMDATE "09May02" +#define DATE "10 May 2002" +#define LSMDATE "10May02" /* Debug flags */ #define DEBUG 1