in Job, Client, and Pool, with precedence such that Job overrides Client
which in turn overrides Pool.
- Print a message when a job starts if the conf file is not current.
-- To pass Include 1 or two letter commands
- I Name Include name - first record
- B Name Base name - repeat
- R "xxx" Regexp
- W "xxx" Wild Card
- E zzz Exclude expression (wild card)
- P "plugin" Plugin
- D "reader" Reader program
- T "writer" Writer program
- O Options In current commpressed format
- (compression, signature, onefs, recurse, sparse,
- replace, verify options, ...)
- N End option set
- B BaseName Start second option set any letter
- ...
- E
- F Number Number of filenames to follow
- <file-name>
- B Name
- ...
- N End option set
- F Number Number of filenames to follow
- <file-name>
- ...
- Spooling ideas taken from Volker Sauer's and other's emails:
> IMHO job spooling should be turned on
VolSessionId and VolSessionTime.
=========================================================
-=============================================================
-
- File Backup Options Project
- 28 April 2004
-
-
-Subject: File Backup Options
-
-Problem:
- Bacula users who are using compression would like to suppress
- compressing all .gz files since it was a waste of CPU time.
- Although Bacula currently permits using different options
- (compression, ...) on a directory by directory basis, it cannot
- do it on a file by file basis, which is clearly what was
- desired.
-
-Proposed Implementation:
- To solve this problem, I propose the following:
-
- - Add a new Director resource type called Options.
-
- - The Options resource will have records for all options that
- can currently be specified on the Include record (in a
- FileSet). Examples below.
-
- - The Options resource will permit an exclude option as well
- as a number of additional options.
-
- - The heart of the Options resource is the ability to supply
- any number of Regex records which specify POSIX regular
- expressions. These regular expressions are applied to the
- fully qualified filename (path and all). If one matches,
- then the Options will be used.
-
- - When a Regex specification matches an included file, the
- options specified in the Options resource will override the
- default options specified on the Include record.
-
- - Include records will be modified to permit referencing one or
- more Options resources. The Options will be used in the
- order listed on the Include record and the first one that
- matches will be applied.
-
- - Options (or specifications) currently supplied on the Include
- record will be deprecated (i.e. removed in a later version a
- year or so from now).
-
- - The Exclude record will be deprecated as the same functionality
- can be obtained by using an Exclude = yes in the Options or
- an Exclude = xxxx directive.
-
-Options records:
- The following records can appear in the Options resource. An
- asterisk preceding the name indicates a feature not currently
- implemented.
-
- - Regexp "xxx" - Match regular expression
- - Wild "xxx" - Do a wild card match
-
- For Backup Jobs:
- - Compression= (GZIP, ...)
- - Signature= (MD5, SHA1, ...)
- - *Encryption=
- - OneFs= (yes/no) - remain on one filesystem
- - Recurse= (yes/no) - recurse into subdirectories
- - Sparse= (yes/no) - do sparse file backup
- - *Exclude= (yes/no) - exclude file from being saved
- - *INclude= (yes/no) - include the file (needed??)
- - *Reader= (filename) - external read (backup) program
- - *Plugin= (filename) - read/write plugin module
-
- For Verify Jobs:
- - verify= (ipnougsamc5) - verify options
-
- For Restore Jobs:
- - replace= (always/ifnewer/ifolder/never) - replace options currently
- implemented in 1.31
- - *Writer= (filename) - external write (restore) program
-
-
-Implementation:
- Currently options specifying compression, MD5 signatures,
- recursion, ... of a FileSet are supplied on the Include
- record. These will now all be collected into a Options
- resource, which will be specified in the Include in place of
- the options. Multiple Options may be specified. Since the
- Options may contain regular expressions that are applied to the
- full filename, this will give the ability to specify backup
- options on a file by file basis to whatever level of detail you
- wish.
-
-Example:
-
- Today:
-
- FileSet {
- Name = "FullSet"
- Include = compression=GZIP signature=MD5 {
- /
- }
- }
-
- Proposal:
-
- FileSet {
- Name = "FullSet"
- Include {
- Options {
- Compression = GZIP;
- Signature = MD5
- Wild = /*.?*/ # matches all files.
- }
- Include = /
- }
- }
-
- That's a lot more to do the same thing, but it gives the ability to
- apply options on a file by file basis. For example, suppose you
- want to compress all files but not any file with extensions .gz or .Z.
- In that case, you will need to group two sets of options using
- the Options resource. Files may be anywhere except in an
- option set ??? All Options apply to all files in the order
- the Options were specified. To have files included with
- different option sets without using wild-cards, use two or more
- Includes -- each one is handled in turn using only the files and
- optionsets specified in the include.
-
- FileSet {
- Name = "FullSet"
- Include {
- Options {
- Signature = MD5
- # Note multiple Matches are ORed
- Wild = "*.gz" # matches .gz files
- Wild = "*.Z" # matches .Z files
- }
- Options {
- Compression = GZIP
- Signature = MD5
- Wild = "*.?*" # matches all files
- }
- Include = /
- }
- }
-
- Now, since the no Compression option is specified in the
- first group of Options, *.gz or *.Z file will have an MD5 signature computed,
- but will not be compressed. For all other files, the *.gz *.Z will not
- match, so the second group of options will be used which will include GZIP
- compression.
-
-Questions:
-
-Results:
- After implementing the above, the user will be able to specify
- on a file by file basis (using regular expressions) what options are
- applied for the backup.
-
-
-=============================================
-
==========================================================
Unsaved File design
====
-=== Done in 1.33
-- Change console to bconsole.
-- Change smtp to bsmtp.
-- Fix time difference problem between Bacula and Client
- so that everything is in GMT.
-- Fix TimeZone problem!
-- Mount a tape that is not right for the job (wrong # files on tape)
- Bacula asks for another tape, fix problems with first tape and
- say "mount". All works OK, but status shows:
- Device /dev/nst0 open but no Bacula volume is mounted.
- Total Bytes=1,153,820,213 Blocks=17,888 Bytes/block=64,502
- Positioned at File=9 Block=3,951
- Full Backup job Rufus.2003-10-26_16.45.31 using Volume "DLT-24Oct03" on device /dev/nst0
- Files=21,003 Bytes=253,954,408 Bytes/sec=2,919,016
- FDReadSeqNo=192,134 in_msg=129830 out_msg=5 fd=7
-- Upgrade to cygwin 1.5
-- Optimize fsf not to read.
-- Use ioctl() fsf if it exists. Figure out where we are from
- the mt_status command. Use slow fsf only if other does not work.
-- Enhance "update slots" to include a "scan" feature
- scan 1; scan 1-5; scan 1,2,4 ... to update the catalog
-- Allow a slot or range of slots on the label barcodes command.
-- Finish implementation of Verify=DiskToCatalog
-- Make sure that Volumes are recycled based on "Least recently used"
- rather than lowest MediaId.
-- Add flag to write only one EOF mark on the tape.
-- Implement autochanger testing in btape "test" command.
-- Implement lmark to list everyfile marked.
-- Make mark/unmark report how many files marked/unmarked.
-- Keep last 5 or 10 completed jobs and show them in a similar list.
-- Make a Running Jobs: output similar to current Scheduled Jobs:
-- Change "create_media_record in bscan to use Archive instead of Full.
-- Have some way to estimate the restore size or have it printed.
-- Volume problems occurs if you have valid volume, written, then it is
- truncated. You get 12-Nov-2003 11:48 rufus-sd: kernsave.2003-11-12_11.48.09 Warning: mount.c:228 Volume on /tmp is not a Bacula labeled Volume, because:
- block.c:640 Read zero bytes on device /tmp.
-- Make sure that 64 bit I/O packets are used on Cygwin.
-- Add to supported autochangers
- OS : FreeBSD-4.9
- Auto-Changer : QUALSTAR TLS-4210
- Manufufactur : Qualstar
- Tapes : 12 (AIT1: 36GB, AIT2: 50GB all uncompressed)
- Drives : 2xAIT2 (installed in the Qualstar: SONY SDX-500C AIT2)
-- Document estimate command in tree.
-- Document lsmark command in tree.
-- Setup a standard job that builds a bootstrap file and saves
- it with the catalog database.
-- See if a restore job can add a file to the tape (prohibit this).
-- Restrict characters permitted in a name.
-- In restore, provide option for limiting to a particular Pool.
-- In restore, list FileSets that only have different base names --
- i.e. any FileSet with the same name should be treated as the same.
-- Make Scheduler sort jobs by StartTime, Priority.
-- Make sure smtp and any other useful program is executable by the world
- in case Bacula is not running as root.
-- Look at Dan's field width problems in PostgreSQL.
-- Look at effect of removing GROUP BYs.
-- In restore take all filesets with same base name.
-- From Alan Brown <ajb2@mssl.ucl.ac.uk>
- BTW, there's a make install bug in 1.33 - with --enable-gnome,
- gnome-console is built, but the binary and .conf are not being installed.
-- Permit Bacula and apcupsd donations (not done for apcupsd).
-- Fix Ctl-C crashing the Console (readline?).
-- Look at code in recycle_oldes_purged_volume() recycle.c. Why not
- let SQL do ORDER BY LastWritten ASC?
-- Look at find_next_volume() algorithm. Currently, it selects:
- +---------+------------+---------------------+-----------+
- | MediaId | VolumeName | LastWritten | VolBytes |
- +---------+------------+---------------------+-----------+
- | 3 | Test13 | 0000-00-00 00:00:00 | 1 |
- | 4 | Test14 | 0000-00-00 00:00:00 | 1 |
- | 1 | test11 | 2003-12-03 18:39:55 | 4,004,926 |
- | 2 | test12 | 2004-01-04 15:25:56 | 2,078,691 |
- +---------+------------+---------------------+-----------+
- but perhaps it should fill already used Volumes first, and use
- Append volumes before Purged, or Recycled, ...
-- Possibly remove the "|| ap == NULL" on lines 123 and 207 of lib/var.c,
- which creates compile problems on alpha systems.
- var.c:123: no match for `va_list & == long int'
-- Check "restore" 3 (JobId), then it asks for Storage resource. Does
- it verify that the correct volume is chosen?
-- Make Bacula "poll a drive".
-- Notes for final checking of Nic's code:
- Could I get you to double check the switch () statements in the
- job_check_maxwaittime and job_check_maxruntime functions in
- src/dird/job.c?
-- Define week of year for scheduler. W01, W02, ...
- Week 01 of a year is per definition the first week that has the
- Thursday in this year, which is equivalent to the week that contains the
- fourth day of January. In other words, the first week of a new year is
- the week that has the majority of its days in the new year. Week 01
- might also contain days from the previous year and the week before week
- 01 of a year is the last week (52 or 53) of the previous year even if it
- contains days from the new year. A week starts with Monday (day 1) and
- ends with Sunday (day 7). For example, the first week of the year 1997
- lasts from 1996-12-30 to 1997-01-05 and can be written in standard
- notation as
- 1997-W01 or 1997W01
- The week notation can also be extended by a number indicating the day
- of the week. For example, the day 1996-12-31, which is the Tuesday (day
- 2) of the first week of 1997, can also be written as
- 1997-W01-2 or 1997W012
-- Either restrict the characters in a name, or fix the problem
- emailing with names containing / (smtp command line breaks).
-- Implement .consolerc for Console
-- Implement scan: for every slot it finds, zero the slot of
- Volume other volume having that slot.
-- Make restore job check if all the files are actually restored.
-- Look at 2Gb limit for SQLite.
-- Fix get_storage_from_media_type (ua_restore) to use command line
- storage=
-- Don't print "Warning: Wrong Volume mounted ..." if mounting second volume.
-- Write a mini-readline with history and editing.
-- Take a careful look a the Basic recycling algorithm. When Bacula
- chooses, the order should be:
- - Look for Append
- - Look for Recycle or Purged
- - Prune volumes
- - Look for purged
- Instead of using lowest media Id, find the least recently used
- volume.
-
- When the tape is mounted and Bacula requests the status
- - Do everything possible to use it.
-
- Define a "available" status, which is the currently mounted
- Volume and all volumes that are currently in the autochanger.
-- Is a pool specification really needed for a restore? Yes, and
- you may want to exclude archive Pools.
-- Implement a PostgreSQL driver.
-- Fix restore to list errors if Invalid block found, and if # files
- restored does not match # expected.
-- Something is not right in last block of fill command.
-- Add FileSet to command line arguments for restore.
-- Enhance time and size scanning routines.
-- Add Console usr permissions -- do by adding filters for
- jobs, clients, storage, ...
-- Put max network buffer size on a directive.
-- Why does "mark cygwin" take so long!!!!!!!!
-- Implement alist processing for ACLs from Console.
-- When a file is set for restore, walk back up the chain of
- directories, setting them to be restored.
-- Figure out a way to set restore on a directory without recursively
- decending. (recurse off?).
-- Fix restore to only pull in last Differential and later Incrementals.
-- Implement 3 Pools for a Job:
- Job {
- Name = ...
- Full Backup Pool = xxx
- Incremental Backup Pool = yyy
- Differential Backup Pool = zzz
- }
-- Look at ASSERT() at 384 src/lib/bnet.c
-- Dates are wrong in restore list from Win32 FD.
-- Dates are wrong in catalog from Win32 FD.
-- Remove h_errno from bnet.c by including proper header.
-- For "list jobs" order by EndTime.
-- Make two tape fill test work.
-- Add atime preservation.
-- Do not err job if could not write bootstrap file.
-- Save and restore last_job across executions.
-- Have each daemon save the last_jobs structure when exiting and
- read it back in when starting up.
-- "restore jobid=1 select" calls get_storage_xxx, which prints "JobId 1 is
- not running."
-- Make column listing for running jobs JobId Level Type Started Name Status
-- Why does Bacula need the drive open to do "autochanger list" ?
-- Add data compare on write/read in btape "test".
-- Rescue builds incorrect script files on Rufus.
-- Release SQLite 2.8.9
-- During install, copy any console.conf to bconsole.conf.
-- Check: Run = Level=Differential feb-dec 1 at 1:05 to see if wday is empty.
-- Look at installation file permissions with Scott so that make install
- and the rpms agree.
-- Finish code passing files=nnn to restore start.
-- Add ctl-c to console to stop current command and discard buffered
- output.
-- Estimate to Tibs never returns.
-- Symbolic link a directory to another one, then backup the symbolic link.
-- Check and possibly fix problems with hard links.
-- Fix query buffer clobber ua_query.c
-- Allow "delete job jobid=xx jobid=xxx".
-- Update volume FromPool (or FromPool=xxx) refreshes the Volume defaults from Pool.
-- Implement multiple Volume in "purge jobs volume=".
-- Test Qmsg() code to be used in bnet.c to prevent recursion. Queue the
- message. If dequeueing toss the messages. Lock while dequeuing so that
- it cannot be called recursively and set dequeuing flag.
-- On unknown client in restore "client=xxx"
- Could not find Client "Matou": ERR=Query failed: DROP TABLE temp1:
- ERR=no such table: temp1
-- Null dlist head and tail pointers if all items are removed.
-- Build console in client-only build.
-- Phil says that Windows file sizes mismatch in Verify when they should,
- and that either the file size or the catalog size was zero.
-- Check time/dates printed during restore when using Win32 API.
-- Once a job is canceled, make sure the status reflects that even if the
- job is "stuck" in the run queue.
-- Test work on conio.c -- particularly linking.
-- Complete Win32 installer
-- Add ms disk example
+=== Done in 1.35
+- To pass Include 1 or two letter commands
+ I Name Include name - first record
+ B Name Base name - repeat
+ R "xxx" Regexp
+ W "xxx" Wild Card
+ E zzz Exclude expression (wild card)
+ P "plugin" Plugin
+ D "reader" Reader program
+ T "writer" Writer program
+ O Options In current commpressed format
+ (compression, signature, onefs, recurse, sparse,
+ replace, verify options, ...)
+ N End option set
+ B BaseName Start second option set any letter
+ ...
+ E
+ F Number Number of filenames to follow
+ <file-name>
+ B Name
+ ...
+ N End option set
+ F Number Number of filenames to follow
+ <file-name>
+ ...
+
int send_exclude_list(JCR *jcr)
{
BSOCK *fd = jcr->file_bsock;
+ if (jcr->fileset->new_include) {
+ return 1;
+ }
bnet_fsend(fd, exc);
return send_list(jcr, EXC_LIST);
}
*/
static RES_ITEM newinc_items[] = {
{"file", store_fname, NULL, 0, 0, 0},
- {"include", store_fname, NULL, 0, 0, 0},
{"options", options_res, NULL, 0, 0, 0},
{NULL, NULL, NULL, 0, 0, 0}
};
/*
* Decide if we are doing a new Include or an old include. The
- * new Include is followed immediately by {, whereas the
+ * new Include is followed immediately by open brace, whereas the
* old include has options following the Include.
*/
token = lex_get_token(lc, T_ALL);
/*
* Store FileSet FInclude/FExclude info
* Note, when this routine is called, we are inside a FileSet
- * resource. We treat the Finclude/Fexeclude like a sort of
+ * resource. We treat the Include/Execlude like a sort of
* mini-resource within the FileSet resource.
*/
static void store_newinc(LEX *lc, RES_ITEM *item, int index, int pass)
static int runbefore_cmd(JCR *jcr);
static int runafter_cmd(JCR *jcr);
static int run_cmd(JCR *jcr, char *cmd, const char *name);
+static void set_options(findFOPTS *fo, const char *opts);
/* Exported functions */
return true;
}
-static findFOPTS *set_options(FF_PKT *ff)
+static findFOPTS *start_options(FF_PKT *ff)
{
int state = ff->fileset->state;
findINCEXE *incexe = ff->fileset->incexe;
fileset->incexe->name_list.append(bstrdup(item));
break;
case 'R':
- current_opts = set_options(ff);
+ current_opts = start_options(ff);
current_opts->regex.append(bstrdup(item));
state = state_options;
break;
case 'B':
- current_opts = set_options(ff);
+ current_opts = start_options(ff);
current_opts->base.append(bstrdup(item));
state = state_options;
break;
case 'W':
- current_opts = set_options(ff);
+ current_opts = start_options(ff);
current_opts->wild.append(bstrdup(item));
state = state_options;
break;
case 'O':
- current_opts = set_options(ff);
- bstrncpy(current_opts->opts, item, MAX_FOPTS);
+ current_opts = start_options(ff);
+ set_options(current_opts, item);
state = state_options;
break;
default:
Dmsg0(400, "I\n");
for (j=0; j<incexe->opts_list.size(); j++) {
findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
- Dmsg1(400, "O %s\n", fo->opts);
for (k=0; k<fo->regex.size(); k++) {
Dmsg1(400, "R %s\n", (char *)fo->regex.get(k));
}
Dmsg0(400, "E\n");
for (j=0; j<incexe->opts_list.size(); j++) {
findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
- Dmsg1(400, "O %s\n", fo->opts);
for (k=0; k<fo->regex.size(); k++) {
Dmsg1(400, "R %s\n", (char *)fo->regex.get(k));
}
Dmsg1(400, "F %s\n", (char *)incexe->name_list.get(j));
}
}
-
-
return ff->fileset->state != state_error;
}
+/*
+ * As an optimization, we should do this during
+ * "compile" time in filed/job.c, and keep only a bit mask
+ * and the Verify options.
+ */
+static void set_options(findFOPTS *fo, const char *opts)
+{
+ int j;
+ const char *p;
+
+ for (p=opts; *p; p++) {
+ switch (*p) {
+ case 'a': /* alway replace */
+ case '0': /* no option */
+ break;
+ case 'e':
+ fo->flags |= FO_EXCLUDE;
+ break;
+ case 'f':
+ fo->flags |= FO_MULTIFS;
+ break;
+ case 'h': /* no recursion */
+ fo->flags |= FO_NO_RECURSION;
+ break;
+ case 'M': /* MD5 */
+ fo->flags |= FO_MD5;
+ break;
+ case 'n':
+ fo->flags |= FO_NOREPLACE;
+ break;
+ case 'p': /* use portable data format */
+ fo->flags |= FO_PORTABLE;
+ break;
+ case 'r': /* read fifo */
+ fo->flags |= FO_READFIFO;
+ break;
+ case 'S':
+ fo->flags |= FO_SHA1;
+ break;
+ case 's':
+ fo->flags |= FO_SPARSE;
+ break;
+ case 'm':
+ fo->flags |= FO_MTIMEONLY;
+ break;
+ case 'k':
+ fo->flags |= FO_KEEPATIME;
+ break;
+ case 'V': /* verify options */
+ /* Copy Verify Options */
+ for (j=0; *p && *p != ':'; p++) {
+ fo->VerifyOpts[j] = *p;
+ if (j < (int)sizeof(fo->VerifyOpts) - 1) {
+ j++;
+ }
+ }
+ fo->VerifyOpts[j] = 0;
+ break;
+ case 'w':
+ fo->flags |= FO_IF_NEWER;
+ break;
+ case 'Z': /* gzip compression */
+ fo->flags |= FO_GZIP;
+ fo->GZIP_level = *++p - '0';
+ Dmsg1(200, "Compression level=%d\n", fo->GZIP_level);
+ break;
+ default:
+ Emsg1(M_ERROR, 0, "Unknown include/exclude option: %c\n", *p);
+ break;
+ }
+ }
+}
+
/*
* Director is passing his Fileset
#endif
} /* End inner switch */
- case FT_DIR:
+ case FT_DIRBEGIN:
+ case FT_DIREND:
Dmsg2(200, "Make dir mode=%o dir=%s\n", new_mode, attr->ofname);
if (make_path(jcr, attr->ofname, new_mode, parent_mode, uid, gid, 0, NULL) != 0) {
return CF_ERROR;
#include "find.h"
-int32_t name_max; /* filename max length */
-int32_t path_max; /* path name max length */
+int32_t name_max; /* filename max length */
+int32_t path_max; /* path name max length */
/* ****FIXME**** debug until stable */
#undef bmalloc
#define bmalloc(x) sm_malloc(__FILE__, __LINE__, x)
-static void set_options(FF_PKT *ff, const char *opts);
static int our_callback(FF_PKT *ff, void *hpkt);
static bool accept_file(FF_PKT *ff);
*/
FF_PKT *init_find_files()
{
- FF_PKT *ff;
+ FF_PKT *ff;
ff = (FF_PKT *)bmalloc(sizeof(FF_PKT));
memset(ff, 0, sizeof(FF_PKT));
ff->sys_fname = get_pool_memory(PM_FNAME);
- init_include_exclude_files(ff); /* init lists */
+ init_include_exclude_files(ff); /* init lists */
/* Get system path and filename maximum lengths */
path_max = pathconf(".", _PC_PATH_MAX);
if (name_max < 1024) {
name_max = 1024;
}
- path_max++; /* add for EOS */
- name_max++; /* add for EOS */
+ path_max++; /* add for EOS */
+ name_max++; /* add for EOS */
Dmsg1(100, "init_find_files ff=%p\n", ff);
return ff;
findFILESET *fileset = ff->fileset;
if (fileset) {
int i, j;
+ ff->flags = 0;
+ ff->VerifyOpts[0] = 0;
for (i=0; i<fileset->include_list.size(); i++) {
- findINCEXE *incexe = (findINCEXE *)fileset->include_list.get(i);
- fileset->incexe = incexe;
- /*
- * By setting all options, we in effect or the global options
- * which is what we want.
- */
- for (j=0; j<incexe->opts_list.size(); j++) {
- findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
- Dmsg1(400, "Find global options O %s\n", fo->opts);
- set_options(ff, fo->opts);
- }
- for (j=0; j<incexe->name_list.size(); j++) {
+ findINCEXE *incexe = (findINCEXE *)fileset->include_list.get(i);
+ fileset->incexe = incexe;
+ /*
+ * By setting all options, we in effect or the global options
+ * which is what we want.
+ */
+ for (j=0; j<incexe->opts_list.size(); j++) {
+ findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
+ ff->flags |= fo->flags;
+ ff->GZIP_level = fo->GZIP_level;
+ bstrncpy(ff->VerifyOpts, fo->VerifyOpts, sizeof(ff->VerifyOpts));
+ }
+ for (j=0; j<incexe->name_list.size(); j++) {
Dmsg1(400, "F %s\n", (char *)incexe->name_list.get(j));
- char *fname = (char *)incexe->name_list.get(j);
- if (!find_one_file(jcr, ff, our_callback, his_pkt, fname, (dev_t)-1, 1)) {
- return 0; /* error return */
- }
- }
+ char *fname = (char *)incexe->name_list.get(j);
+ if (!find_one_file(jcr, ff, our_callback, his_pkt, fname, (dev_t)-1, 1)) {
+ return 0; /* error return */
+ }
+ }
}
} else {
struct s_included_file *inc = NULL;
/* This is the old deprecated way */
while (!job_canceled(jcr) && (inc = get_next_included_file(ff, inc))) {
- /* Copy options for this file */
- bstrncpy(ff->VerifyOpts, inc->VerifyOpts, sizeof(ff->VerifyOpts));
+ /* Copy options for this file */
+ bstrncpy(ff->VerifyOpts, inc->VerifyOpts, sizeof(ff->VerifyOpts));
Dmsg1(50, "find_files: file=%s\n", inc->fname);
- if (!file_is_excluded(ff, inc->fname)) {
- if (!find_one_file(jcr, ff, callback, his_pkt, inc->fname, (dev_t)-1, 1)) {
- return 0; /* error return */
- }
- }
+ if (!file_is_excluded(ff, inc->fname)) {
+ if (!find_one_file(jcr, ff, callback, his_pkt, inc->fname, (dev_t)-1, 1)) {
+ return 0; /* error return */
+ }
+ }
}
}
return 1;
for (j=0; j<incexe->opts_list.size(); j++) {
findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
for (k=0; k<fo->wild.size(); k++) {
-
+ if (fnmatch((char *)fo->wild.get(k), ff->fname, 0) == 0) {
+ ff->flags = fo->flags;
+ ff->GZIP_level = fo->GZIP_level;
+ if (ff->flags & FO_EXCLUDE) {
+ return false; /* reject file */
+ }
+ return true; /* accept file */
+ }
}
}
for (i=0; i<fileset->exclude_list.size(); i++) {
findINCEXE *incexe = (findINCEXE *)fileset->exclude_list.get(i);
for (j=0; j<incexe->opts_list.size(); j++) {
- findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
- for (k=0; k<fo->wild.size(); k++) {
- Dmsg1(400, "W %s\n", (char *)fo->wild.get(k));
- if (fnmatch((char *)fo->wild.get(k), ff->fname, FNM_PATHNAME) == 0) {
- Dmsg1(000, "Reject wild: %s\n", ff->fname);
- return false; /* reject file */
- }
- }
+ findFOPTS *fo = (findFOPTS *)incexe->opts_list.get(j);
+ for (k=0; k<fo->wild.size(); k++) {
+ if (fnmatch((char *)fo->wild.get(k), ff->fname, 0) == 0) {
+ Dmsg1(400, "Reject wild1: %s\n", ff->fname);
+ return false; /* reject file */
+ }
+ }
}
for (j=0; j<incexe->name_list.size(); j++) {
- Dmsg1(400, "F %s\n", (char *)incexe->name_list.get(j));
- if (fnmatch((char *)incexe->name_list.get(j), ff->fname, FNM_PATHNAME) == 0) {
- Dmsg1(000, "Reject: %s\n", ff->fname);
- return false; /* reject file */
- }
+ if (fnmatch((char *)incexe->name_list.get(j), ff->fname, 0) == 0) {
+ Dmsg1(400, "Reject wild2: %s\n", ff->fname);
+ return false; /* reject file */
+ }
}
}
- Dmsg1(000, "Accept: %s\n", ff->fname);
return true;
}
case FT_NORECURSE:
case FT_NOFSCHG:
case FT_NOOPEN:
- Dmsg1(000, "File=%s\n", ff->fname);
return ff->callback(ff, hpkt);
/* These items can be filtered */
case FT_DIREND:
case FT_SPEC:
if (accept_file(ff)) {
- return ff->callback(ff, hpkt);
+ return ff->callback(ff, hpkt);
} else {
- return 0;
+ return 0;
}
- }
+ }
return 0;
}
-/*
- * As an optimization, we should do this during
- * "compile" time in filed/job.c, and keep only a bit mask
- * and the Verify options.
- */
-static void set_options(FF_PKT *ff, const char *opts)
-{
- int j;
- const char *p;
-
- for (p=opts; *p; p++) {
- switch (*p) {
- case 'a': /* alway replace */
- case '0': /* no option */
- break;
- case 'e':
- ff->flags |= FO_EXCLUDE;
- break;
- case 'f':
- ff->flags |= FO_MULTIFS;
- break;
- case 'h': /* no recursion */
- ff->flags |= FO_NO_RECURSION;
- break;
- case 'M': /* MD5 */
- ff->flags |= FO_MD5;
- break;
- case 'n':
- ff->flags |= FO_NOREPLACE;
- break;
- case 'p': /* use portable data format */
- ff->flags |= FO_PORTABLE;
- break;
- case 'r': /* read fifo */
- ff->flags |= FO_READFIFO;
- break;
- case 'S':
- ff->flags |= FO_SHA1;
- break;
- case 's':
- ff->flags |= FO_SPARSE;
- break;
- case 'm':
- ff->flags |= FO_MTIMEONLY;
- break;
- case 'k':
- ff->flags |= FO_KEEPATIME;
- break;
- case 'V': /* verify options */
- /* Copy Verify Options */
- for (j=0; *p && *p != ':'; p++) {
- ff->VerifyOpts[j] = *p;
- if (j < (int)sizeof(ff->VerifyOpts) - 1) {
- j++;
- }
- }
- ff->VerifyOpts[j] = 0;
- break;
- case 'w':
- ff->flags |= FO_IF_NEWER;
- break;
- case 'Z': /* gzip compression */
- ff->flags |= FO_GZIP;
- ff->GZIP_level = *++p - '0';
- Dmsg1(200, "Compression level=%d\n", ff->GZIP_level);
- break;
- default:
- Emsg1(M_ERROR, 0, "Unknown include/exclude option: %c\n", *p);
- break;
- }
- }
-}
-
-
/*
* Terminate find_files() and release
* all allocated memory
/* File options structure */
struct findFOPTS {
- char opts[MAX_FOPTS]; /* options string */
uint32_t flags; /* options in bits */
int GZIP_level; /* GZIP level */
char VerifyOpts[MAX_FOPTS]; /* verify options */
#include "bacula.h"
#include "find.h"
-extern int32_t name_max; /* filename max length */
-extern int32_t path_max; /* path name max length */
+extern int32_t name_max; /* filename max length */
+extern int32_t path_max; /* path name max length */
/*
* Structure for keeping track of hard linked files, we
*/
struct f_link {
struct f_link *next;
- dev_t dev; /* device */
- ino_t ino; /* inode with device is unique */
+ dev_t dev; /* device */
+ ino_t ino; /* inode with device is unique */
short linkcount;
- uint32_t FileIndex; /* Bacula FileIndex of this file */
- char name[1]; /* The name */
+ uint32_t FileIndex; /* Bacula FileIndex of this file */
+ char name[1]; /* The name */
};
static void free_dir_ff_pkt(FF_PKT *dir_ff_pkt)
}
/*
- * Find a single file.
+ * Find a single file.
* handle_file is the callback for handling the file.
* p is the filename
* parent_device is the device we are currently on
*/
int
find_one_file(JCR *jcr, FF_PKT *ff_pkt, int handle_file(FF_PKT *ff, void *hpkt),
- void *pkt, char *fname, dev_t parent_device, int top_level)
+ void *pkt, char *fname, dev_t parent_device, int top_level)
{
struct utimbuf restore_times;
int rtn_stat;
Dmsg1(300, "Non-directory incremental: %s\n", ff_pkt->fname);
/* Not a directory */
if (ff_pkt->statp.st_mtime < ff_pkt->save_time
- && ((ff_pkt->flags & FO_MTIMEONLY) ||
- ff_pkt->statp.st_ctime < ff_pkt->save_time)) {
- /* Incremental option, file not changed */
- ff_pkt->type = FT_NOCHG;
- return handle_file(ff_pkt, pkt);
+ && ((ff_pkt->flags & FO_MTIMEONLY) ||
+ ff_pkt->statp.st_ctime < ff_pkt->save_time)) {
+ /* Incremental option, file not changed */
+ ff_pkt->type = FT_NOCHG;
+ return handle_file(ff_pkt, pkt);
}
}
*/
if (ff_pkt->statp.st_nlink > 1
&& (S_ISREG(ff_pkt->statp.st_mode)
- || S_ISCHR(ff_pkt->statp.st_mode)
- || S_ISBLK(ff_pkt->statp.st_mode)
- || S_ISFIFO(ff_pkt->statp.st_mode)
- || S_ISSOCK(ff_pkt->statp.st_mode))) {
+ || S_ISCHR(ff_pkt->statp.st_mode)
+ || S_ISBLK(ff_pkt->statp.st_mode)
+ || S_ISFIFO(ff_pkt->statp.st_mode)
+ || S_ISSOCK(ff_pkt->statp.st_mode))) {
struct f_link *lp;
/* Search link list of hard linked files */
for (lp = ff_pkt->linklist; lp; lp = lp->next)
- if (lp->ino == ff_pkt->statp.st_ino && lp->dev == ff_pkt->statp.st_dev) {
+ if (lp->ino == ff_pkt->statp.st_ino && lp->dev == ff_pkt->statp.st_dev) {
/* If we have already backed up the hard linked file don't do it again */
- if (strcmp(lp->name, fname) == 0) {
+ if (strcmp(lp->name, fname) == 0) {
Jmsg1(jcr, M_WARNING, 0, _("Attempt to backup hard linked file %s twice ignored.\n"),
- fname);
- return 1; /* ignore */
- }
- ff_pkt->link = lp->name;
- ff_pkt->type = FT_LNKSAVED; /* Handle link, file already saved */
- ff_pkt->LinkFI = lp->FileIndex;
- return handle_file(ff_pkt, pkt);
- }
+ fname);
+ return 1; /* ignore */
+ }
+ ff_pkt->link = lp->name;
+ ff_pkt->type = FT_LNKSAVED; /* Handle link, file already saved */
+ ff_pkt->LinkFI = lp->FileIndex;
+ return handle_file(ff_pkt, pkt);
+ }
/* File not previously dumped. Chain it into our list. */
lp = (struct f_link *)bmalloc(sizeof(struct f_link) + strlen(fname) +1);
strcpy(lp->name, fname);
lp->next = ff_pkt->linklist;
ff_pkt->linklist = lp;
- ff_pkt->linked = lp; /* mark saved link */
+ ff_pkt->linked = lp; /* mark saved link */
} else {
ff_pkt->linked = NULL;
}
sizeleft = ff_pkt->statp.st_size;
/* Don't bother opening empty, world readable files. Also do not open
- files when archive is meant for /dev/null. */
+ files when archive is meant for /dev/null. */
if (ff_pkt->null_output_device || (sizeleft == 0
- && MODE_RALL == (MODE_RALL & ff_pkt->statp.st_mode))) {
- ff_pkt->type = FT_REGE;
+ && MODE_RALL == (MODE_RALL & ff_pkt->statp.st_mode))) {
+ ff_pkt->type = FT_REGE;
} else {
- ff_pkt->type = FT_REG;
+ ff_pkt->type = FT_REG;
}
rtn_stat = handle_file(ff_pkt, pkt);
if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
}
return rtn_stat;
size = readlink(fname, buffer, path_max + name_max + 101);
if (size < 0) {
- /* Could not follow link */
- ff_pkt->type = FT_NOFOLLOW;
- ff_pkt->ff_errno = errno;
- rtn_stat = handle_file(ff_pkt, pkt);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- return rtn_stat;
+ /* Could not follow link */
+ ff_pkt->type = FT_NOFOLLOW;
+ ff_pkt->ff_errno = errno;
+ rtn_stat = handle_file(ff_pkt, pkt);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ return rtn_stat;
}
buffer[size] = 0;
- ff_pkt->link = buffer; /* point to link */
- ff_pkt->type = FT_LNK; /* got a real link */
+ ff_pkt->link = buffer; /* point to link */
+ ff_pkt->type = FT_LNK; /* got a real link */
rtn_stat = handle_file(ff_pkt, pkt);
if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
}
return rtn_stat;
struct dirent *entry, *result;
char *link;
int link_len;
- int len;
+ int len;
int status;
dev_t our_device = ff_pkt->statp.st_dev;
* in principle, we should be able to access everything.
*/
if (!have_win32_api() || (ff_pkt->flags & FO_PORTABLE)) {
- if (access(fname, R_OK) == -1 && geteuid() != 0) {
- /* Could not access() directory */
- ff_pkt->type = FT_NOACCESS;
- ff_pkt->ff_errno = errno;
- rtn_stat = handle_file(ff_pkt, pkt);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- return rtn_stat;
- }
+ if (access(fname, R_OK) == -1 && geteuid() != 0) {
+ /* Could not access() directory */
+ ff_pkt->type = FT_NOACCESS;
+ ff_pkt->ff_errno = errno;
+ rtn_stat = handle_file(ff_pkt, pkt);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ return rtn_stat;
+ }
}
/* Build a canonical directory name with a trailing slash in link var */
bstrncpy(link, fname, link_len);
/* Strip all trailing slashes */
while (len >= 1 && link[len - 1] == '/')
- len--;
+ len--;
link[len++] = '/'; /* add back one */
link[len] = 0;
ff_pkt->link = link;
if (ff_pkt->incremental &&
- (ff_pkt->statp.st_mtime < ff_pkt->save_time &&
- ff_pkt->statp.st_ctime < ff_pkt->save_time)) {
- /* Incremental option, directory entry not changed */
- ff_pkt->type = FT_DIRNOCHG;
+ (ff_pkt->statp.st_mtime < ff_pkt->save_time &&
+ ff_pkt->statp.st_ctime < ff_pkt->save_time)) {
+ /* Incremental option, directory entry not changed */
+ ff_pkt->type = FT_DIRNOCHG;
} else {
- ff_pkt->type = FT_DIRBEGIN;
+ ff_pkt->type = FT_DIRBEGIN;
}
/* Send off Directory packet now */
if (!handle_file(ff_pkt, pkt)) {
- free(link);
- return 0; /* Do not save this directory */
+ free(link);
+ return 0; /* Do not save this directory */
}
if (ff_pkt->type == FT_DIRBEGIN) {
- ff_pkt->type = FT_DIREND;
+ ff_pkt->type = FT_DIREND;
}
/*
dir_ff_pkt->excluded_files_list = NULL;
dir_ff_pkt->excluded_paths_list = NULL;
dir_ff_pkt->linklist = NULL;
-
+
ff_pkt->link = ff_pkt->fname; /* reset "link" */
/*
* user has turned it off for this directory.
*/
if (ff_pkt->flags & FO_NO_RECURSION) {
- /* No recursion into this directory */
- ff_pkt->type = FT_NORECURSE;
- rtn_stat = handle_file(ff_pkt, pkt);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- free(link);
- free_dir_ff_pkt(dir_ff_pkt);
- return rtn_stat;
+ /* No recursion into this directory */
+ ff_pkt->type = FT_NORECURSE;
+ rtn_stat = handle_file(ff_pkt, pkt);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ free(link);
+ free_dir_ff_pkt(dir_ff_pkt);
+ return rtn_stat;
}
/*
* avoid doing so if the user only wants to dump one file system.
*/
if (!top_level && !(ff_pkt->flags & FO_MULTIFS) &&
- parent_device != ff_pkt->statp.st_dev) {
- /* returning here means we do not handle this directory */
- ff_pkt->type = FT_NOFSCHG;
- rtn_stat = handle_file(ff_pkt, pkt);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- free(link);
- free_dir_ff_pkt(dir_ff_pkt);
- return rtn_stat;
+ parent_device != ff_pkt->statp.st_dev) {
+ /* returning here means we do not handle this directory */
+ ff_pkt->type = FT_NOFSCHG;
+ rtn_stat = handle_file(ff_pkt, pkt);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ free(link);
+ free_dir_ff_pkt(dir_ff_pkt);
+ return rtn_stat;
}
/*
* Decend into or "recurse" into the directory to read
*/
errno = 0;
if ((directory = opendir(fname)) == NULL) {
- ff_pkt->type = FT_NOOPEN;
- ff_pkt->ff_errno = errno;
- rtn_stat = handle_file(ff_pkt, pkt);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- free(link);
- free_dir_ff_pkt(dir_ff_pkt);
- return rtn_stat;
+ ff_pkt->type = FT_NOOPEN;
+ ff_pkt->ff_errno = errno;
+ rtn_stat = handle_file(ff_pkt, pkt);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ free(link);
+ free_dir_ff_pkt(dir_ff_pkt);
+ return rtn_stat;
}
/*
rtn_stat = 1;
entry = (struct dirent *)malloc(sizeof(struct dirent) + name_max + 100);
for ( ; !job_canceled(jcr); ) {
- char *p, *q;
- int i;
+ char *p, *q;
+ int i;
- status = readdir_r(directory, entry, &result);
- if (status != 0 || result == NULL) {
+ status = readdir_r(directory, entry, &result);
+ if (status != 0 || result == NULL) {
// Dmsg2(99, "readdir returned stat=%d result=0x%x\n",
-// status, (long)result);
- break;
- }
- ASSERT(name_max+1 > (int)sizeof(struct dirent) + (int)NAMELEN(entry));
- p = entry->d_name;
+// status, (long)result);
+ break;
+ }
+ ASSERT(name_max+1 > (int)sizeof(struct dirent) + (int)NAMELEN(entry));
+ p = entry->d_name;
/* Skip `.', `..', and excluded file names. */
if (p[0] == '\0' || (p[0] == '.' && (p[1] == '\0' ||
(p[1] == '.' && p[2] == '\0')))) {
- continue;
- }
-
- if ((int)NAMELEN(entry) + len >= link_len) {
- link_len = len + NAMELEN(entry) + 1;
- link = (char *)brealloc(link, link_len + 1);
- }
- q = link + len;
- for (i=0; i < (int)NAMELEN(entry); i++) {
- *q++ = *p++;
- }
- *q = 0;
- if (!file_is_excluded(ff_pkt, link)) {
- rtn_stat = find_one_file(jcr, ff_pkt, handle_file, pkt, link, our_device, 0);
- if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
- }
- }
+ continue;
+ }
+
+ if ((int)NAMELEN(entry) + len >= link_len) {
+ link_len = len + NAMELEN(entry) + 1;
+ link = (char *)brealloc(link, link_len + 1);
+ }
+ q = link + len;
+ for (i=0; i < (int)NAMELEN(entry); i++) {
+ *q++ = *p++;
+ }
+ *q = 0;
+ if (!file_is_excluded(ff_pkt, link)) {
+ rtn_stat = find_one_file(jcr, ff_pkt, handle_file, pkt, link, our_device, 0);
+ if (ff_pkt->linked) {
+ ff_pkt->linked->FileIndex = ff_pkt->FileIndex;
+ }
+ }
}
closedir(directory);
free(link);
* the directory modes and dates. Temp directory values
* were used without this record.
*/
- handle_file(dir_ff_pkt, pkt); /* handle directory entry */
+ handle_file(dir_ff_pkt, pkt); /* handle directory entry */
if (ff_pkt->linked) {
- ff_pkt->linked->FileIndex = dir_ff_pkt->FileIndex;
+ ff_pkt->linked->FileIndex = dir_ff_pkt->FileIndex;
}
free_dir_ff_pkt(dir_ff_pkt);
if (ff_pkt->flags & FO_KEEPATIME) {
- utime(fname, &restore_times);
+ utime(fname, &restore_times);
}
return rtn_stat;
} /* end check for directory */
#ifdef HAVE_FREEBSD_OS
/*
* On FreeBSD, all block devices are character devices, so
- * to be able to read a raw disk, we need the check for
- * a character device.
+ * to be able to read a raw disk, we need the check for
+ * a character device.
* crw-r----- 1 root operator - 116, 0x00040002 Jun 9 19:32 /dev/ad0s3
* crw-r----- 1 root operator - 116, 0x00040002 Jun 9 19:32 /dev/rad0s3
*/
#else
if (top_level && S_ISBLK(ff_pkt->statp.st_mode)) {
#endif
- ff_pkt->type = FT_RAW; /* raw partition */
+ ff_pkt->type = FT_RAW; /* raw partition */
} else if (top_level && S_ISFIFO(ff_pkt->statp.st_mode) &&
- ff_pkt->flags & FO_READFIFO) {
+ ff_pkt->flags & FO_READFIFO) {
ff_pkt->type = FT_FIFO;
} else {
/* The only remaining types are special (character, ...) files */
lc = lp;
lp = lp->next;
if (lc) {
- free(lc);
- count++;
+ free(lc);
+ count++;
}
}
return count;
/* Match from the beginning of a component only */
if ((p == file || (*p != '/' && *(p-1) == '/'))
&& file_in_excluded_list(ff->excluded_files_list, p)) {
- return 1;
+ return 1;
}
}
return 0;
projects / bacula / bacula / commitdiff