4 \chapter{The Old FileSet Resource}
5 \label{OldFileSetChapter}
6 \label{FileSetResource}
7 \index[general]{Resource!Old FileSet }
8 \index[general]{Old FileSet Resource }
10 Note, this form of the FileSet resource still works but has been replaced by a
11 new more flexible form in Bacula version 1.34.3. As a consequence, you are
12 encouraged to convert to the new form as this one is deprecated and will be
13 removed in a future version.
15 The FileSet resource defines what files are to be included in a backup job. At
16 least one {\bf FileSet} resource is required. It consists of a list of files
17 or directories to be included, a list of files or directories to be excluded
18 and the various backup options such as compression, encryption, and signatures
19 that are to be applied to each file.
21 Any change to the list of the included files will cause Bacula to
22 automatically create a new FileSet (defined by the name and an MD5 checksum of
23 the Include contents). Each time a new FileSet is created, Bacula will ensure
24 that the first backup is always a Full save.
30 Start of the FileSet records. At least one {\bf FileSet} resource must be
33 \item {\bf Name = \lt{}name\gt{}}
35 The name of the FileSet resource. This record is required.
37 \item {\bf Include = \lt{}processing-options\gt{}
38 \ \ \{ \lt{}file-list\gt{} \}
42 The Include resource specifies the list of files and/or directories to be
43 included in the backup job. There can be any number of {\bf Include} {\bf
44 file-list} specifications within the FileSet, each having its own set of {\bf
45 processing-options}. Normally, the {\bf file-list} consists of one file or
46 directory name per line. Directory names should be specified without a
47 trailing slash. Wild-card (or glob matching) does not work when used in an
48 Include list. It does work in an Exclude list though. Just the same, any
49 asterisk (*), question mark (?), or left-bracket ([) must be preceded by a
50 slash (\textbackslash{}\textbackslash{}) if you want it to represent the
53 You should {\bf always} specify a full path for every directory and file that
54 you list in the FileSet. In addition, on Windows machines, you should {\bf
55 always} prefix the directory or filename with the drive specification (e.g.
56 {\bf c:/xxx}) using Unix directory name separators (forward slash). However,
57 within an {\bf Exclude} where for some reason the exclude will not work with a
58 prefixed drive letter. If you want to specify a drive letter in exclusions on
59 Win32 systems, you can do so by specifying:
63 Exclude = { /cygdrive/d/archive/Mulberry }
67 where in this case, the {\bf /cygdrive/d} \&nsbp; is Cygwin's way of referring
68 to drives on Win32 (thanks to Mathieu Arnold for this tip).
70 Bacula's default for processing directories is to recursively descend in the
71 directory saving all files and subdirectories. Bacula will not by default
72 cross file systems (or mount points in Unix parlance). This means that if you
73 specify the root partition (e.g. {\bf /}), Bacula will save only the root
74 partition and not any of the other mounted file systems. Similarly on Windows
75 systems, you must explicitly specify each of the drives you want saved (e.g.
76 {\bf c:/} and {\bf d:/} ...). In addition, at least for Windows systems, you
77 will most likely want to enclose each specification within double quotes. The
78 {\bf df} command on Unix systems will show you which mount points you must
79 specify to save everything. See below for an example.
81 Take special care not to include a directory twice or Bacula will backup the
82 same files two times wasting a lot of space on your archive device. Including
83 a directory twice is very easy to do. For example:
91 on a Unix system where /usr is a subdirectory (rather than a mounted
92 filesystem) will cause /usr to be backed up twice. In this case, on Bacula
93 versions prior to 1.32f-5-09Mar04 due to a bug, you will not be able to
94 restore hard linked files that were backed up twice.
96 The {\bf \lt{}processing-options\gt{}} is optional. If specified, it is a list
97 of {\bf keyword=value} options to be applied to the file-list. Multiple
98 options may be specified by separating them with spaces. These options are
99 used to modify the default processing behavior of the files included. Since
100 there can be multiple {\bf Include} sets, this permits effectively specifying
101 the desired options (compression, encryption, ...) on a file by file basis.
102 The options may be one of the following:
106 \item {\bf compression=GZIP}
107 \index[fd]{compression }
108 All files saved will be software compressed using the GNU ZIP compression
109 format. The compression is done on a file by file basis by the File daemon.
110 If there is a problem reading the tape in a single record of a file, it will
111 at most affect that file and none of the other files on the tape. Normally
112 this option is {\bf not} needed if you have a modern tape drive as the drive
113 will do its own compression. However, compression is very important if you
114 are writing your Volumes to a file, and it can also be helpful if you have a
115 fast computer but a slow network.
117 Specifying {\bf GZIP} uses the default compression level six (i.e. {\bf GZIP}
118 is identical to {\bf GZIP6}). If you want a different compression level (1
119 through 9), you can specify it by appending the level number with no
120 intervening spaces to {\bf GZIP}. Thus {\bf compression=GZIP1} would give
121 minimum compression but the fastest algorithm, and {\bf compression=GZIP9}
122 would give the highest level of compression, but requires more computation.
123 According to the GZIP documentation, compression levels greater than 6
124 generally give very little extra compression but are rather CPU intensive.
126 \item {\bf signature=MD5}
127 \index[fd]{signature }
128 An MD5 signature will be computed for all files saved. Adding this option
129 generates about 5\% extra overhead for each file saved. In addition to the
130 additional CPU time, the MD5 signature adds 16 more bytes per file to your
131 catalog. We strongly recommend that this option be specified as a default
134 \item {\bf signature=SHA1}
135 \index[fd]{signature }
136 An SHA1 signature will be computed for all The SHA1 algorithm is purported to
137 be some what slower than the MD5 algorithm, but at the same time is
138 significantly better from a cryptographic point of view (i.e. much fewer
139 collisions, much lower probability of being hacked.) It adds four more bytes
140 than the MD5 signature. We strongly recommend that either this option or MD5
141 be specified as a default for all files. Note, only one of the two options
142 MD5 or SHA1 can be computed for any file.
144 \item {\bf *encryption=\lt{}algorithm\gt{}}
145 \index[fd]{*encryption }
146 All files saved will be encrypted using one of the following algorithms (NOT
155 \item {\bf verify=\lt{}options\gt{}}
157 The options letters specified are used when running a {\bf Verify
158 Level=Catalog} job, and may be any combination of the following:
166 compare the permission bits
169 compare the number of links
181 compare the access time
184 compare the modification time (st\_mtime)
187 compare the change time (st\_ctime)
190 report file size decreases
193 compare the MD5 signature
196 compare the SHA1 signature
199 A useful set of general options on the {\bf Level=Catalog} verify is {\bf
200 pins5} i.e. compare permission bits, inodes, number of links, size, and MD5
203 \item {\bf onefs=yes|no}
205 If set to {\bf yes} (the default), {\bf Bacula} will remain on a single file
206 system. That is it will not backup file systems that are mounted on a
207 subdirectory. In this case, you must explicitly list each file system you
208 want saved. If you set this option to {\bf no}, Bacula will backup all
209 mounted file systems (i.e. traverse mount points) that are found within the
210 {\bf FileSet}. Thus if you have NFS or Samba file systems mounted on a
211 directory included in your FileSet, they will also be backed up. Normally, it
212 is preferable to set {\bf onefs=yes} and to explicitly name each file system
213 you want backed up. See the example below for more details.
216 \item {\bf portable=yes|no}
217 \index[fd]{portable }
218 If set to {\bf yes} (default is {\bf no}), the Bacula File daemon will backup
219 Win32 files in a portable format. By default, this option is set to {\bf
220 no}, which means that on Win32 systems, the data will be backed up using
221 Windows API calls and on WinNT/2K/XP, the security and ownership data will be
222 properly backed up (and restored), but the data format is not portable to
223 other systems -- e.g. Unix, Win95/98/Me. On Unix systems, this option is
224 ignored, and unless you have a specific need to have portable backups, we
225 recommend accept the default ({\bf no}) so that the maximum information
226 concerning your files is backed up.
228 \item {\bf recurse=yes|no}
230 If set to {\bf yes} (the default), Bacula will recurse (or descend) into all
231 subdirectories found unless the directory is explicitly excluded using an
232 {\bf exclude} definition. If you set {\bf recurse=no}, Bacula will save the
233 subdirectory entries, but not descend into the subdirectories, and thus will
234 not save the contents of the subdirectories. Normally, you will want the
237 \item {\bf sparse=yes|no}
239 Enable special code that checks for sparse files such as created by ndbm. The
240 default is {\bf no}, so no checks are made for sparse files. You may specify
241 {\bf sparse=yes} even on files that are not sparse file. No harm will be
242 done, but there will be a small additional overhead to check for buffers of
243 all zero, and a small additional amount of space on the output archive will
244 be used to save the seek address of each non-zero record read.
246 {\bf Restrictions:} Bacula reads files in 32K buffers. If the whole buffer is
247 zero, it will be treated as a sparse block and not written to tape. However,
248 if any part of the buffer is non-zero, the whole buffer will be written to
249 tape, possibly including some disk sectors (generally 4098 bytes) that are
250 all zero. As a consequence, Bacula's detection of sparse blocks is in 32K
251 increments rather than the system block size. If anyone considers this to be
252 a real problem, please send in a request for change with the reason. The
253 sparse code was first implemented in version 1.27.
255 If you are not familiar with sparse files, an example is say a file where you
256 wrote 512 bytes at address zero, then 512 bytes at address 1 million. The
257 operating system will allocate only two blocks, and the empty space or hole
258 will have nothing allocated. However, when you read the sparse file and read
259 the addresses where nothing was written, the OS will return all zeros as if
260 the space were allocated, and if you backup such a file, a lot of space will
261 be used to write zeros to the volume. Worse yet, when you restore the file,
262 all the previously empty space will now be allocated using much more disk
263 space. By turning on the {\bf sparse} option, Bacula will specifically look
264 for empty space in the file, and any empty space will not be written to the
265 Volume, nor will it be restored. The price to pay for this is that Bacula
266 must search each block it reads before writing it. On a slow system, this may
267 be important. If you suspect you have sparse files, you should benchmark the
268 difference or set sparse for only those files that are really sparse.
271 \item {\bf readfifo=yes|no}
272 \index[fd]{readfifo }
273 If enabled, tells the Client to read the data on a backup and write the data
274 on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet.
275 In this case, you must have a program already running that writes into the
276 FIFO for a backup or reads from the FIFO on a restore. This can be
277 accomplished with the {\bf RunBeforeJob} record. If this is not the case,
278 Bacula will hang indefinitely on reading/writing the FIFO. When this is not
279 enabled (default), the Client simply saves the directory entry for the FIFO.
281 \item {\bf mtimeonly=yes|no}
282 \index[dir]{mtimeonly }
283 If enabled, tells the Client that the selection of files during Incremental
284 and Differential backups should based only on the st\_mtime value in the
285 stat() packet. The default is {\bf no} which means that the selection of
286 files to be backed up will be based on both the st\_mtime and the st\_ctime
287 values. In general, it is not recommended to use this option.
289 \item {\bf keepatime=yes|no}
290 \index[dir]{keepatime }
291 The default is {\bf no}. When enabled, Bacula will reset the st\_atime
292 (access time) field of files that it backs up to their value prior to the
293 backup. This option is not generally recommended as there are very few
294 programs that use st\_atime, and the backup overhead is increased because of
295 the additional system call necessary to reset the times. (I'm not sure this
299 {\bf \lt{}file-list\gt{}} is a space separated list of filenames and/or
300 directory names. To include names containing spaces, enclose the name between
301 double-quotes. The list may span multiple lines, in fact, normally it is good
302 practice to specify each filename on a separate line.
304 There are a number of special cases when specifying files or directories in a
305 {\bf file-list}. They are:
308 \item Any file-list item preceded by an at-sign (@) is assumed to be a
309 filename containing a list of files, which is read when the configuration
310 file is parsed during Director startup. Note, that the file is read on the
311 Director's machine and not on the Client.
312 \item Any file-list item beginning with a vertical bar (|) is assumed to be a
313 program. This program will be executed on the Director's machine at the time
314 the Job starts (not when the Director reads the configuration file), and any
315 output from that program will be assumed to be a list of files or
316 directories, one per line, to be included. This allows you to have a job that
317 for example includes all the local partitions even if you change the
318 partitioning by adding a disk. In general, you will need to prefix your
319 command or commands with a {\bf sh -c} so that they are invoked by a shell.
320 This will not be the case if you are invoking a script as in the second
321 example below. Also, you must take care to escape wild-cards and ensure that
322 any spaces in your command are escaped as well. If you use a single quotes
323 (') within a double quote ("), Bacula will treat everything between the
324 single quotes as one field so it will not be necessary to escape the spaces.
325 In general, getting all the quotes and escapes correct is a real pain as you
326 can see by the next example. As a consequence, it is often easier to put
327 everything in a file, and simply us the file name within Bacula. In that case
328 the {\bf sh -c} will not be necessary providing the first line of the file is
336 Include = signature=SHA1 {
337 "|sh -c 'df -l | grep \"^/dev/hd[ab]\" | grep -v \".*/tmp\" \
338 | awk \"{print \\$6}\"'"
343 will produce a list of all the local partitions on a Red Hat Linux system.
344 Note, the above line was split, but should normally be written on one line.
345 Quoting is a real problem because you must quote for Bacula which consists of
346 preceding every \textbackslash{} and every " with a \textbackslash{}, and
347 you must also quote for the shell command. In the end, it is probably easier
348 just to execute a small file with:
352 Include = signature=MD5 {
358 where my\_partitions has:
363 df -l | grep "^/dev/hd[ab]" | grep -v ".*/tmp" \
368 If the vertical bar (|) is preceded by a backslash as in \textbackslash{}|,
369 the program will be executed on the Client's machine instead of on the
370 Director's machine -- (this is implemented but not tested, and very likely
371 will not work on Windows).
372 \item Any file-list item preceded by a less-than sign (\lt{}) will be taken
373 to be a file. This file will be read on the Director's machine at the time
374 the Job starts, and the data will be assumed to be a list of directories or
375 files, one per line, to be included. This feature allows you to modify the
376 external file and change what will be saved without stopping and restarting
377 Bacula as would be necessary if using the @ modifier noted above.
379 If you precede the less-than sign (\lt{}) with a backslash as in
380 \textbackslash{}\lt{}, the file-list will be read on the Client machine
381 instead of on the Director's machine (implemented but not tested).
382 \item If you explicitly specify a block device such as {\bf /dev/hda1}, then
383 Bacula (starting with version 1.28) will assume that this is a raw partition
384 to be backed up. In this case, you are strongly urged to specify a {\bf
385 sparse=yes} include option, otherwise, you will save the whole partition
386 rather than just the actual data that the partition contains. For example:
390 Include = signature=MD5 sparse=yes {
396 will backup the data in device /dev/hd6.
398 Ludovic Strappazon has pointed out that this feature can be used to backup a
399 full Microsoft Windows disk. Simply boot into the system using a Linux Rescue
400 disk, then load a statically linked Bacula as described in the
401 \ilink{ Disaster Recovery Using Bacula}{RescueChapter} chapter of
402 this manual. Then simply save the whole disk partition. In the case of a
403 disaster, you can then restore the desired partition.
404 \item If you explicitly specify a FIFO device name (created with mkfifo), and
405 you add the option {\bf readfifo=yes} as an option, Bacula will read the FIFO
406 and back its data up to the Volume. For example:
410 Include = signature=SHA1 readfifo=yes {
416 if {\bf /home/abc/fifo} is a fifo device, Bacula will open the fifo, read it,
417 and store all data thus obtained on the Volume. Please note, you must have a
418 process on the system that is writing into the fifo, or Bacula will hang, and
419 after one minute of waiting, it will go on to the next file. The data read
420 can be anything since Bacula treats it as a stream.
422 This feature can be an excellent way to do a "hot" backup of a very large
423 database. You can use the {\bf RunBeforeJob} to create the fifo and to start
424 a program that dynamically reads your database and writes it to the fifo.
425 Bacula will then write it to the Volume.
427 During the restore operation, the inverse is true, after Bacula creates the
428 fifo if there was any data stored with it (no need to explicitly list it or
429 add any options), that data will be written back to the fifo. As a
430 consequence, if any such FIFOs exist in the fileset to be restored, you must
431 ensure that there is a reader program or Bacula will block, and after one
432 minute, Bacula will time out the write to the fifo and move on to the next
436 The Exclude Files specifies the list of files and/or directories to be
437 excluded from the backup job. The {\bf \lt{}file-list\gt{}} is a comma or
438 space separated list of filenames and/or directory names. To exclude names
439 containing spaces, enclose the name between double-quotes. Most often each
440 filename is on a separate line.
442 For exclusions on Windows systems, do not include a leading drive letter such
443 as {\bf c:}. This does not work. Any filename preceded by an at-sign (@) is
444 assumed to be a filename on the Director's machine containing a list of files.
448 The following is an example of a valid FileSet resource definition:
454 Include = compression=GZIP signature=SHA1 sparse=yes {
459 /usr/lib/another_file
466 Note, in the above example, all the files contained in /etc/backup.list will
467 be compressed with GZIP compression, an SHA1 signature will be computed on the
468 file's contents (its data), and sparse file handling will apply.
470 The two files /root/myfile and /usr/lib/another\_file will also be saved but
471 without any options. In addition, all files with the extension {\bf .o} will
472 be excluded from the file set (i.e. from the backup).
474 Suppose you want to save everything except {\bf /tmp} on your system. Doing a
475 {\bf df} command, you get the following output:
480 Filesystem 1k-blocks Used Available Use% Mounted on
481 /dev/hda5 5044156 439232 4348692 10% /
482 /dev/hda1 62193 4935 54047 9% /boot
483 /dev/hda9 20161172 5524660 13612372 29% /home
484 /dev/hda2 62217 6843 52161 12% /rescue
485 /dev/hda8 5044156 42548 4745376 1% /tmp
486 /dev/hda6 5044156 2613132 2174792 55% /usr
487 none 127708 0 127708 0% /dev/shm
488 //minimatou/c$ 14099200 9895424 4203776 71% /mnt/mmatou
489 lmatou:/ 1554264 215884 1258056 15% /mnt/matou
490 lmatou:/home 2478140 1589952 760072 68% /mnt/matou/home
491 lmatou:/usr 1981000 1199960 678628 64% /mnt/matou/usr
492 lpmatou:/ 995116 484112 459596 52% /mnt/pmatou
493 lpmatou:/home 19222656 2787880 15458228 16% /mnt/pmatou/home
494 lpmatou:/usr 2478140 2038764 311260 87% /mnt/pmatou/usr
495 deuter:/ 4806936 97684 4465064 3% /mnt/deuter
496 deuter:/home 4806904 280100 4282620 7% /mnt/deuter/home
497 deuter:/files 44133352 27652876 14238608 67% /mnt/deuter/files
501 Now, if you specify only {\bf /} in your Include list, Bacula will only save
502 the Filesystem {\bf /dev/hda5}. To save all file systems except {\bf /tmp}
503 with out including any of the Samba or NFS mounted systems, and explicitly
504 excluding a /tmp, /proc, .journal, and .autofsck, which you will not want to
505 be saved and restored, you can use the following:
528 Since /tmp is on its own filesystem and it was not explicitly named in the
529 Include list, it is not really needed in the exclude list. It is better to
530 list it in the Exclude list for clarity, and in case the disks are changed so
531 that it is no longer in its own partition.
533 Please be aware that allowing Bacula to traverse or change file systems can be
534 {\bf very} dangerous. For example, with the following:
547 you will be backing up an NFS mounted partition ({\bf /mnt/matou}), and since
548 {\bf onefs} is set to {\bf no}, Bacula will traverse file systems. However, if
549 {\bf /mnt/matou} has the current machine's file systems mounted, as is often
550 the case, you will get yourself into a recursive loop and the backup will
553 The following FileSet definition will backup a raw partition:
558 Name = "RawPartition"
559 Include = sparse=yes {
566 Note, in backing up and restoring a raw partition, you should ensure that no
567 other process including the system is writing to that partition. As a
568 precaution, you are strongly urged to ensure that the raw partition is not
569 mounted or is mounted read-only. If necessary, this can be done using the {\bf
570 RunBeforeJob} record.
572 \section{Additional Considerations for Using Excludes on Windows}
573 \index[general]{Additional Considerations for Using Excludes on Windows }
574 \index[general]{Windows!Additional Considerations for Using Excludes on }
577 For exclude lists to work correctly on Windows, you must observe the following
581 \item Filenames are case sensitive, so you must use the correct case.
582 \item To exclude a directory, you must not have a trailing slash on the
584 \item If you have spaces in your filename, you must enclose the entire name
585 in double-quote characters ("). Trying to use a backslash before the space
587 \item You must not precede the excluded file or directory with a drive letter
588 (such as {\bf c:}) otherwise it will not work.
591 Thanks to Thiago Lima for summarizing the above items for us. If you are
592 having difficulties getting includes or excludes to work, you might want to
593 try using the {\bf estimate job=xxx listing} command documented in the
594 \ilink{Console chapter}{console.tex#estimate} of this manual.
597 \section{Windows Considerations for FileSets}
598 \index[general]{FileSets!Windows Considerations for }
599 \index[general]{Windows Considerations for FileSets }
601 If you are entering Windows file names, the directory path may be preceded by
602 the drive and a colon (as in c:). However, the path separators must be
603 specified in Unix convention (i.e. forward slash (/)). If you wish to include
604 a quote in a file name, precede the quote with a backslash
605 (\textbackslash{}\textbackslash{}). For example you might use the following
606 for a Windows machine to backup the "My Documents" directory:
615 Exclude = { *.obj *.exe }
620 When using exclusion on Windows, do not use a drive prefix (i.e. {\bf c:}) as
621 it will prevent the exclusion from working. However, if you need to specify a
622 drive letter in exclusions on Win32 systems, you can do so by specifying:
626 Exclude = { /cygdrive/d/archive/Mulberry }
630 where in this case, the {\bf /cygdrive/d} is Cygwin's way of referring to
631 drive {\bf d:} (thanks to Mathieu Arnold for this tip).
633 \section{A Windows Example FileSet}
634 \index[general]{FileSet!Windows Example }
635 \index[general]{Windows Example FileSet }
637 The following example was contributed by Phil Stracchino:
641 This is my Windows 2000 fileset:
643 Name = "Windows 2000 Full Set"
644 Include = signature=MD5 {
647 # Most of these files are excluded not because we don't want
648 # them, but because Win2K won't allow them to be backed up
649 # except via proprietary Win32 API calls.
651 "/Documents and Settings/*/Application Data/*/Profiles/*/*/
653 "/Documents and Settings/*/Local Settings/Application Data/
654 Microsoft/Windows/[Uu][Ss][Rr][Cc][Ll][Aa][Ss][Ss].*"
655 "/Documents and Settings/*/[Nn][Tt][Uu][Ss][Ee][Rr].*"
656 "/Documents and Settings/*/Cookies/*"
657 "/Documents and Settings/*/Local Settings/History/*"
658 "/Documents and Settings/*/Local Settings/
659 Temporary Internet Files/*"
660 "/Documents and Settings/*/Local Settings/Temp/*"
662 "/WINNT/security/logs/scepol.log"
663 "/WINNT/system32/config/*"
664 "/WINNT/msdownld.tmp/*"
665 "/WINNT/Internet Logs/*"
666 "/WINNT/$Nt*Uninstall*"
676 Note, the three line of the above Exclude were split to fit on the document
677 page, they should be written on a single line in real use.