4 \section*{The Old FileSet Resource}
6 \label{FileSetResource}
7 \index[general]{Resource!Old FileSet }
8 \index[general]{Old FileSet Resource }
9 \addcontentsline{toc}{section}{Old FileSet Resource}
11 Note, this form of the FileSet resource still works but has been replaced by a
12 new more flexible form in Bacula version 1.34.3. As a consequence, you are
13 encouraged to convert to the new form as this one is deprecated and will be
14 removed in a future version.
16 The FileSet resource defines what files are to be included in a backup job. At
17 least one {\bf FileSet} resource is required. It consists of a list of files
18 or directories to be included, a list of files or directories to be excluded
19 and the various backup options such as compression, encryption, and signatures
20 that are to be applied to each file.
22 Any change to the list of the included files will cause Bacula to
23 automatically create a new FileSet (defined by the name and an MD5 checksum of
24 the Include contents). Each time a new FileSet is created, Bacula will ensure
25 that the first backup is always a Full save.
31 Start of the FileSet records. At least one {\bf FileSet} resource must be
34 \item {\bf Name = \lt{}name\gt{}}
36 The name of the FileSet resource. This record is required.
38 \item {\bf Include = \lt{}processing-options\gt{}
39 \ \ \{ \lt{}file-list\gt{} \}
43 The Include resource specifies the list of files and/or directories to be
44 included in the backup job. There can be any number of {\bf Include} {\bf
45 file-list} specifications within the FileSet, each having its own set of {\bf
46 processing-options}. Normally, the {\bf file-list} consists of one file or
47 directory name per line. Directory names should be specified without a
48 trailing slash. Wild-card (or glob matching) does not work when used in an
49 Include list. It does work in an Exclude list though. Just the same, any
50 asterisk (*), question mark (?), or left-bracket ([) must be preceded by a
51 slash (\textbackslash{}\textbackslash{}) if you want it to represent the
54 You should {\bf always} specify a full path for every directory and file that
55 you list in the FileSet. In addition, on Windows machines, you should {\bf
56 always} prefix the directory or filename with the drive specification (e.g.
57 {\bf c:/xxx}) using Unix directory name separators (forward slash). However,
58 within an {\bf Exclude} where for some reason the exclude will not work with a
59 prefixed drive letter. If you want to specify a drive letter in exclusions on
60 Win32 systems, you can do so by specifying:
64 Exclude = { /cygdrive/d/archive/Mulberry }
68 where in this case, the {\bf /cygdrive/d} \&nsbp; is Cygwin's way of referring
69 to drives on Win32 (thanks to Mathieu Arnold for this tip).
71 Bacula's default for processing directories is to recursively descend in the
72 directory saving all files and subdirectories. Bacula will not by default
73 cross file systems (or mount points in Unix parlance). This means that if you
74 specify the root partition (e.g. {\bf /}), Bacula will save only the root
75 partition and not any of the other mounted file systems. Similarly on Windows
76 systems, you must explicitly specify each of the drives you want saved (e.g.
77 {\bf c:/} and {\bf d:/} ...). In addition, at least for Windows systems, you
78 will most likely want to enclose each specification within double quotes. The
79 {\bf df} command on Unix systems will show you which mount points you must
80 specify to save everything. See below for an example.
82 Take special care not to include a directory twice or Bacula will backup the
83 same files two times wasting a lot of space on your archive device. Including
84 a directory twice is very easy to do. For example:
92 on a Unix system where /usr is a subdirectory (rather than a mounted
93 filesystem) will cause /usr to be backed up twice. In this case, on Bacula
94 versions prior to 1.32f-5-09Mar04 due to a bug, you will not be able to
95 restore hard linked files that were backed up twice.
97 The {\bf \lt{}processing-options\gt{}} is optional. If specified, it is a list
98 of {\bf keyword=value} options to be applied to the file-list. Multiple
99 options may be specified by separating them with spaces. These options are
100 used to modify the default processing behavior of the files included. Since
101 there can be multiple {\bf Include} sets, this permits effectively specifying
102 the desired options (compression, encryption, ...) on a file by file basis.
103 The options may be one of the following:
107 \item {\bf compression=GZIP}
108 \index[fd]{compression }
109 All files saved will be software compressed using the GNU ZIP compression
110 format. The compression is done on a file by file basis by the File daemon.
111 If there is a problem reading the tape in a single record of a file, it will
112 at most affect that file and none of the other files on the tape. Normally
113 this option is {\bf not} needed if you have a modern tape drive as the drive
114 will do its own compression. However, compression is very important if you
115 are writing your Volumes to a file, and it can also be helpful if you have a
116 fast computer but a slow network.
118 Specifying {\bf GZIP} uses the default compression level six (i.e. {\bf GZIP}
119 is identical to {\bf GZIP6}). If you want a different compression level (1
120 through 9), you can specify it by appending the level number with no
121 intervening spaces to {\bf GZIP}. Thus {\bf compression=GZIP1} would give
122 minimum compression but the fastest algorithm, and {\bf compression=GZIP9}
123 would give the highest level of compression, but requires more computation.
124 According to the GZIP documentation, compression levels greater than 6
125 generally give very little extra compression but are rather CPU intensive.
127 \item {\bf signature=MD5}
128 \index[fd]{signature }
129 An MD5 signature will be computed for all files saved. Adding this option
130 generates about 5\% extra overhead for each file saved. In addition to the
131 additional CPU time, the MD5 signature adds 16 more bytes per file to your
132 catalog. We strongly recommend that this option be specified as a default
135 \item {\bf signature=SHA1}
136 \index[fd]{signature }
137 An SHA1 signature will be computed for all The SHA1 algorithm is purported to
138 be some what slower than the MD5 algorithm, but at the same time is
139 significantly better from a cryptographic point of view (i.e. much fewer
140 collisions, much lower probability of being hacked.) It adds four more bytes
141 than the MD5 signature. We strongly recommend that either this option or MD5
142 be specified as a default for all files. Note, only one of the two options
143 MD5 or SHA1 can be computed for any file.
145 \item {\bf *encryption=\lt{}algorithm\gt{}}
146 \index[fd]{*encryption }
147 All files saved will be encrypted using one of the following algorithms (NOT
156 \item {\bf verify=\lt{}options\gt{}}
158 The options letters specified are used when running a {\bf Verify
159 Level=Catalog} job, and may be any combination of the following:
167 compare the permission bits
170 compare the number of links
182 compare the access time
185 compare the modification time (st\_mtime)
188 compare the change time (st\_ctime)
191 report file size decreases
194 compare the MD5 signature
197 compare the SHA1 signature
200 A useful set of general options on the {\bf Level=Catalog} verify is {\bf
201 pins5} i.e. compare permission bits, inodes, number of links, size, and MD5
204 \item {\bf onefs=yes|no}
206 If set to {\bf yes} (the default), {\bf Bacula} will remain on a single file
207 system. That is it will not backup file systems that are mounted on a
208 subdirectory. In this case, you must explicitly list each file system you
209 want saved. If you set this option to {\bf no}, Bacula will backup all
210 mounted file systems (i.e. traverse mount points) that are found within the
211 {\bf FileSet}. Thus if you have NFS or Samba file systems mounted on a
212 directory included in your FileSet, they will also be backed up. Normally, it
213 is preferable to set {\bf onefs=yes} and to explicitly name each file system
214 you want backed up. See the example below for more details.
217 \item {\bf portable=yes|no}
218 \index[fd]{portable }
219 If set to {\bf yes} (default is {\bf no}), the Bacula File daemon will backup
220 Win32 files in a portable format. By default, this option is set to {\bf
221 no}, which means that on Win32 systems, the data will be backed up using
222 Windows API calls and on WinNT/2K/XP, the security and ownership data will be
223 properly backed up (and restored), but the data format is not portable to
224 other systems -- e.g. Unix, Win95/98/Me. On Unix systems, this option is
225 ignored, and unless you have a specific need to have portable backups, we
226 recommend accept the default ({\bf no}) so that the maximum information
227 concerning your files is backed up.
229 \item {\bf recurse=yes|no}
231 If set to {\bf yes} (the default), Bacula will recurse (or descend) into all
232 subdirectories found unless the directory is explicitly excluded using an
233 {\bf exclude} definition. If you set {\bf recurse=no}, Bacula will save the
234 subdirectory entries, but not descend into the subdirectories, and thus will
235 not save the contents of the subdirectories. Normally, you will want the
238 \item {\bf sparse=yes|no}
240 Enable special code that checks for sparse files such as created by ndbm. The
241 default is {\bf no}, so no checks are made for sparse files. You may specify
242 {\bf sparse=yes} even on files that are not sparse file. No harm will be
243 done, but there will be a small additional overhead to check for buffers of
244 all zero, and a small additional amount of space on the output archive will
245 be used to save the seek address of each non-zero record read.
247 {\bf Restrictions:} Bacula reads files in 32K buffers. If the whole buffer is
248 zero, it will be treated as a sparse block and not written to tape. However,
249 if any part of the buffer is non-zero, the whole buffer will be written to
250 tape, possibly including some disk sectors (generally 4098 bytes) that are
251 all zero. As a consequence, Bacula's detection of sparse blocks is in 32K
252 increments rather than the system block size. If anyone considers this to be
253 a real problem, please send in a request for change with the reason. The
254 sparse code was first implemented in version 1.27.
256 If you are not familiar with sparse files, an example is say a file where you
257 wrote 512 bytes at address zero, then 512 bytes at address 1 million. The
258 operating system will allocate only two blocks, and the empty space or hole
259 will have nothing allocated. However, when you read the sparse file and read
260 the addresses where nothing was written, the OS will return all zeros as if
261 the space were allocated, and if you backup such a file, a lot of space will
262 be used to write zeros to the volume. Worse yet, when you restore the file,
263 all the previously empty space will now be allocated using much more disk
264 space. By turning on the {\bf sparse} option, Bacula will specifically look
265 for empty space in the file, and any empty space will not be written to the
266 Volume, nor will it be restored. The price to pay for this is that Bacula
267 must search each block it reads before writing it. On a slow system, this may
268 be important. If you suspect you have sparse files, you should benchmark the
269 difference or set sparse for only those files that are really sparse.
272 \item {\bf readfifo=yes|no}
273 \index[fd]{readfifo }
274 If enabled, tells the Client to read the data on a backup and write the data
275 on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet.
276 In this case, you must have a program already running that writes into the
277 FIFO for a backup or reads from the FIFO on a restore. This can be
278 accomplished with the {\bf RunBeforeJob} record. If this is not the case,
279 Bacula will hang indefinitely on reading/writing the FIFO. When this is not
280 enabled (default), the Client simply saves the directory entry for the FIFO.
282 \item {\bf mtimeonly=yes|no}
283 \index[dir]{mtimeonly }
284 If enabled, tells the Client that the selection of files during Incremental
285 and Differential backups should based only on the st\_mtime value in the
286 stat() packet. The default is {\bf no} which means that the selection of
287 files to be backed up will be based on both the st\_mtime and the st\_ctime
288 values. In general, it is not recommended to use this option.
290 \item {\bf keepatime=yes|no}
291 \index[dir]{keepatime }
292 The default is {\bf no}. When enabled, Bacula will reset the st\_atime
293 (access time) field of files that it backs up to their value prior to the
294 backup. This option is not generally recommended as there are very few
295 programs that use st\_atime, and the backup overhead is increased because of
296 the additional system call necessary to reset the times. (I'm not sure this
300 {\bf \lt{}file-list\gt{}} is a space separated list of filenames and/or
301 directory names. To include names containing spaces, enclose the name between
302 double-quotes. The list may span multiple lines, in fact, normally it is good
303 practice to specify each filename on a separate line.
305 There are a number of special cases when specifying files or directories in a
306 {\bf file-list}. They are:
309 \item Any file-list item preceded by an at-sign (@) is assumed to be a
310 filename containing a list of files, which is read when the configuration
311 file is parsed during Director startup. Note, that the file is read on the
312 Director's machine and not on the Client.
313 \item Any file-list item beginning with a vertical bar (|) is assumed to be a
314 program. This program will be executed on the Director's machine at the time
315 the Job starts (not when the Director reads the configuration file), and any
316 output from that program will be assumed to be a list of files or
317 directories, one per line, to be included. This allows you to have a job that
318 for example includes all the local partitions even if you change the
319 partitioning by adding a disk. In general, you will need to prefix your
320 command or commands with a {\bf sh -c} so that they are invoked by a shell.
321 This will not be the case if you are invoking a script as in the second
322 example below. Also, you must take care to escape wild-cards and ensure that
323 any spaces in your command are escaped as well. If you use a single quotes
324 (') within a double quote ("), Bacula will treat everything between the
325 single quotes as one field so it will not be necessary to escape the spaces.
326 In general, getting all the quotes and escapes correct is a real pain as you
327 can see by the next example. As a consequence, it is often easier to put
328 everything in a file, and simply us the file name within Bacula. In that case
329 the {\bf sh -c} will not be necessary providing the first line of the file is
337 Include = signature=SHA1 {
338 "|sh -c 'df -l | grep \"^/dev/hd[ab]\" | grep -v \".*/tmp\" \
339 | awk \"{print \\$6}\"'"
344 will produce a list of all the local partitions on a RedHat Linux system.
345 Note, the above line was split, but should normally be written on one line.
346 Quoting is a real problem because you must quote for Bacula which consists of
347 preceding every \textbackslash{} and every " with a \textbackslash{}, and
348 you must also quote for the shell command. In the end, it is probably easier
349 just to execute a small file with:
353 Include = signature=MD5 {
359 where my\_partitions has:
364 df -l | grep "^/dev/hd[ab]" | grep -v ".*/tmp" \
369 If the vertical bar (|) is preceded by a backslash as in \textbackslash{}|,
370 the program will be executed on the Client's machine instead of on the
371 Director's machine -- (this is implemented but not tested, and very likely
372 will not work on Windows).
373 \item Any file-list item preceded by a less-than sign (\lt{}) will be taken
374 to be a file. This file will be read on the Director's machine at the time
375 the Job starts, and the data will be assumed to be a list of directories or
376 files, one per line, to be included. This feature allows you to modify the
377 external file and change what will be saved without stopping and restarting
378 Bacula as would be necessary if using the @ modifier noted above.
380 If you precede the less-than sign (\lt{}) with a backslash as in
381 \textbackslash{}\lt{}, the file-list will be read on the Client machine
382 instead of on the Director's machine (implemented but not tested).
383 \item If you explicitly specify a block device such as {\bf /dev/hda1}, then
384 Bacula (starting with version 1.28) will assume that this is a raw partition
385 to be backed up. In this case, you are strongly urged to specify a {\bf
386 sparse=yes} include option, otherwise, you will save the whole partition
387 rather than just the actual data that the partition contains. For example:
391 Include = signature=MD5 sparse=yes {
397 will backup the data in device /dev/hd6.
399 Ludovic Strappazon has pointed out that this feature can be used to backup a
400 full Microsoft Windows disk. Simply boot into the system using a Linux Rescue
401 disk, then load a statically linked Bacula as described in the
402 \ilink{ Disaster Recovery Using Bacula}{rescue.tex#_ChapterStart} chapter of
403 this manual. Then simply save the whole disk partition. In the case of a
404 disaster, you can then restore the desired partition.
405 \item If you explicitly specify a FIFO device name (created with mkfifo), and
406 you add the option {\bf readfifo=yes} as an option, Bacula will read the FIFO
407 and back its data up to the Volume. For example:
411 Include = signature=SHA1 readfifo=yes {
417 if {\bf /home/abc/fifo} is a fifo device, Bacula will open the fifo, read it,
418 and store all data thus obtained on the Volume. Please note, you must have a
419 process on the system that is writing into the fifo, or Bacula will hang, and
420 after one minute of waiting, it will go on to the next file. The data read
421 can be anything since Bacula treats it as a stream.
423 This feature can be an excellent way to do a "hot" backup of a very large
424 database. You can use the {\bf RunBeforeJob} to create the fifo and to start
425 a program that dynamically reads your database and writes it to the fifo.
426 Bacula will then write it to the Volume.
428 During the restore operation, the inverse is true, after Bacula creates the
429 fifo if there was any data stored with it (no need to explicitly list it or
430 add any options), that data will be written back to the fifo. As a
431 consequence, if any such FIFOs exist in the fileset to be restored, you must
432 ensure that there is a reader program or Bacula will block, and after one
433 minute, Bacula will time out the write to the fifo and move on to the next
437 The Exclude Files specifies the list of files and/or directories to be
438 excluded from the backup job. The {\bf \lt{}file-list\gt{}} is a comma or
439 space separated list of filenames and/or directory names. To exclude names
440 containing spaces, enclose the name between double-quotes. Most often each
441 filename is on a separate line.
443 For exclusions on Windows systems, do not include a leading drive letter such
444 as {\bf c:}. This does not work. Any filename preceded by an at-sign (@) is
445 assumed to be a filename on the Director's machine containing a list of files.
449 The following is an example of a valid FileSet resource definition:
455 Include = compression=GZIP signature=SHA1 sparse=yes {
460 /usr/lib/another_file
467 Note, in the above example, all the files contained in /etc/backup.list will
468 be compressed with GZIP compression, an SHA1 signature will be computed on the
469 file's contents (its data), and sparse file handling will apply.
471 The two files /root/myfile and /usr/lib/another\_file will also be saved but
472 without any options. In addition, all files with the extension {\bf .o} will
473 be excluded from the file set (i.e. from the backup).
475 Suppose you want to save everything except {\bf /tmp} on your system. Doing a
476 {\bf df} command, you get the following output:
481 Filesystem 1k-blocks Used Available Use% Mounted on
482 /dev/hda5 5044156 439232 4348692 10% /
483 /dev/hda1 62193 4935 54047 9% /boot
484 /dev/hda9 20161172 5524660 13612372 29% /home
485 /dev/hda2 62217 6843 52161 12% /rescue
486 /dev/hda8 5044156 42548 4745376 1% /tmp
487 /dev/hda6 5044156 2613132 2174792 55% /usr
488 none 127708 0 127708 0% /dev/shm
489 //minimatou/c$ 14099200 9895424 4203776 71% /mnt/mmatou
490 lmatou:/ 1554264 215884 1258056 15% /mnt/matou
491 lmatou:/home 2478140 1589952 760072 68% /mnt/matou/home
492 lmatou:/usr 1981000 1199960 678628 64% /mnt/matou/usr
493 lpmatou:/ 995116 484112 459596 52% /mnt/pmatou
494 lpmatou:/home 19222656 2787880 15458228 16% /mnt/pmatou/home
495 lpmatou:/usr 2478140 2038764 311260 87% /mnt/pmatou/usr
496 deuter:/ 4806936 97684 4465064 3% /mnt/deuter
497 deuter:/home 4806904 280100 4282620 7% /mnt/deuter/home
498 deuter:/files 44133352 27652876 14238608 67% /mnt/deuter/files
502 Now, if you specify only {\bf /} in your Include list, Bacula will only save
503 the Filesystem {\bf /dev/hda5}. To save all file systems except {\bf /tmp}
504 with out including any of the Samba or NFS mounted systems, and explicitly
505 excluding a /tmp, /proc, .journal, and .autofsck, which you will not want to
506 be saved and restored, you can use the following:
529 Since /tmp is on its own filesystem and it was not explicitly named in the
530 Include list, it is not really needed in the exclude list. It is better to
531 list it in the Exclude list for clarity, and in case the disks are changed so
532 that it is no longer in its own partition.
534 Please be aware that allowing Bacula to traverse or change file systems can be
535 {\bf very} dangerous. For example, with the following:
548 you will be backing up an NFS mounted partition ({\bf /mnt/matou}), and since
549 {\bf onefs} is set to {\bf no}, Bacula will traverse file systems. However, if
550 {\bf /mnt/matou} has the current machine's file systems mounted, as is often
551 the case, you will get yourself into a recursive loop and the backup will
554 The following FileSet definition will backup a raw partition:
559 Name = "RawPartition"
560 Include = sparse=yes {
567 Note, in backing up and restoring a raw partition, you should ensure that no
568 other process including the system is writing to that partition. As a
569 precaution, you are strongly urged to ensure that the raw partition is not
570 mounted or is mounted read-only. If necessary, this can be done using the {\bf
571 RunBeforeJob} record.
573 \subsection*{Additional Considerations for Using Excludes on Windows}
574 \index[general]{Additional Considerations for Using Excludes on Windows }
575 \index[general]{Windows!Additional Considerations for Using Excludes on }
576 \addcontentsline{toc}{subsection}{Additional Considerations for Using Excludes
579 For exclude lists to work correctly on Windows, you must observe the following
583 \item Filenames are case sensitive, so you must use the correct case.
584 \item To exclude a directory, you must not have a trailing slash on the
586 \item If you have spaces in your filename, you must enclose the entire name
587 in double-quote characters ("). Trying to use a backslash before the space
589 \item You must not precede the excluded file or directory with a drive letter
590 (such as {\bf c:}) otherwise it will not work.
593 Thanks to Thiago Lima for summarizing the above items for us. If you are
594 having difficulties getting includes or excludes to work, you might want to
595 try using the {\bf estimate job=xxx listing} command documented in the
596 \ilink{Console chapter}{console.tex#estimate} of this manual.
599 \subsection*{Windows Considerations for FileSets}
600 \index[general]{FileSets!Windows Considerations for }
601 \index[general]{Windows Considerations for FileSets }
602 \addcontentsline{toc}{subsection}{Windows Considerations for FileSets}
604 If you are entering Windows file names, the directory path may be preceded by
605 the drive and a colon (as in c:). However, the path separators must be
606 specified in Unix convention (i.e. forward slash (/)). If you wish to include
607 a quote in a file name, precede the quote with a backslash
608 (\textbackslash{}\textbackslash{}). For example you might use the following
609 for a Windows machine to backup the "My Documents" directory:
618 Exclude = { *.obj *.exe }
623 When using exclusion on Windows, do not use a drive prefix (i.e. {\bf c:}) as
624 it will prevent the exclusion from working. However, if you need to specify a
625 drive letter in exclusions on Win32 systems, you can do so by specifying:
629 Exclude = { /cygdrive/d/archive/Mulberry }
633 where in this case, the {\bf /cygdrive/d} is Cygwin's way of referring to
634 drive {\bf d:} (thanks to Mathieu Arnold for this tip).
636 \subsection*{A Windows Example FileSet}
637 \index[general]{FileSet!Windows Example }
638 \index[general]{Windows Example FileSet }
639 \addcontentsline{toc}{subsection}{Windows Example FileSet}
641 The following example was contributed by Phil Stracchino:
645 This is my Windows 2000 fileset:
647 Name = "Windows 2000 Full Set"
648 Include = signature=MD5 {
651 # Most of these files are excluded not because we don't want
652 # them, but because Win2K won't allow them to be backed up
653 # except via proprietary Win32 API calls.
655 "/Documents and Settings/*/Application Data/*/Profiles/*/*/
657 "/Documents and Settings/*/Local Settings/Application Data/
658 Microsoft/Windows/[Uu][Ss][Rr][Cc][Ll][Aa][Ss][Ss].*"
659 "/Documents and Settings/*/[Nn][Tt][Uu][Ss][Ee][Rr].*"
660 "/Documents and Settings/*/Cookies/*"
661 "/Documents and Settings/*/Local Settings/History/*"
662 "/Documents and Settings/*/Local Settings/
663 Temporary Internet Files/*"
664 "/Documents and Settings/*/Local Settings/Temp/*"
666 "/WINNT/security/logs/scepol.log"
667 "/WINNT/system32/config/*"
668 "/WINNT/msdownld.tmp/*"
669 "/WINNT/Internet Logs/*"
670 "/WINNT/$Nt*Uninstall*"
680 Note, the three line of the above Exclude were split to fit on the document
681 page, they should be written on a single line in real use.