5 \label{NewFeaturesChapter}
6 \index[general]{New Features}
8 This chapter presents the new features added to the development 2.5.x
9 versions to be released as Bacula version 3.0.0 near the end of 2008.
12 \index[general]{Accurate Backup}
13 As with most other backup programs, Bacula decides what files to backup
14 for Incremental and Differental backup by comparing the change (st\_ctime)
15 and modification (st\_mtime) times of the file to the time the last backup completed.
16 If one of those two times is later than the last backup time, then the file
17 will be backed up. This does not, however, permit tracking what files have
18 been deleted and will miss any file with an old time that may have been
19 restored or moved on the client filesystem.
21 If the {\bf Accurate = \lt{}yes|no\gt{}} directive is enabled (default no) in the
22 Job resource, the job will be run as an Accurate Job. For a {\bf Full}
23 backup, there is no difference, but for {\bf Differential} and {\bf Incremental}
24 backups, the Director will send a list of all previous files backed up, and the
25 File daemon will use that list to determine if any new files have been added or
26 or moved and if any files have been deleted. This allows Bacula to make an accurate
27 backup of your system to that point in time so that if you do a restore, it
28 will restore your system exactly. One note of caution about using Accurate backup is that
29 it requires more resources (CPU and memory) on both the Director and
30 the Client machines to create the list of previous files backed up, to send that
31 list to the File daemon, for the File daemon to keep the list (possibly very big)
32 in memory, and for the File daemon to do comparisons between every file in the
37 \index[general]{Copy Jobs}
38 A new {\bf Copy} job type has been implemented. It is essentially
39 identical to the existing Migration feature with the exception that
40 the Job that is copied is left unchanged. This essentially creates
41 two identical copies of the same backup. The Copy Job runs without
42 using the File daemon by copying the data from the old backup Volume to
43 a different Volume in a different Pool. See the Migration documentation
44 for additional details.
46 \section{Virtual Backup (Vbackup)}
47 \index[general]{Virtual Backup}
48 \index[general]{Vbackup}
50 Bacula's virtual backup feature is often called Synthetic Backup or
51 Consolidation in other backup products. It permits you to consolidate
52 the previous Full backup plus the most recent Differential backup and any
53 subsequent Incremental backups into a new Full backup. This is accomplished
54 without contacting the client by reading the previous backup data and
55 writing it to a volume in a different pool.
57 In some respects the Vbackup feature works similar to a Migration job, in
58 that Bacula normally reads the data from the pool specified in the
59 Job resource, and writes it to the {\bf Next Pool} specified in the
60 Job resource. The input Storage resource and the Output Storage resource
63 The Vbackup is enabled on a Job by Job in the Job resource by specifying
64 a level of {\bf VirtualFull}.
66 A typical Job resource definition might look like the following:
80 # Default pool definition
84 Recycle = yes # Automatically recycle Volumes
85 AutoPrune = yes # Prune expired volumes
86 Volume Retention = 365d # one year
94 Recycle = yes # Automatically recycle Volumes
95 AutoPrune = yes # Prune expired volumes
96 Volume Retention = 365d # one year
100 # Definition of file storage device
107 Maximum Concurrent Jobs = 5
110 # Definition of DDS Virtual tape disk storage device
113 Address = localhost # N.B. Use a fully qualified name here
116 Media Type = DiskChangerMedia
117 Maximum Concurrent Jobs = 4
122 Then in bconsole or via a Run schedule, you would run the job as:
125 run job=MyBackup level=Full
126 run job=MyBackup level=Incremental
127 run job=MyBackup level=Differential
128 run job=MyBackup level=Incremental
129 run job=MyBackup level=Incremental
132 So providing there were changes between each of those jobs, you would end up
133 with a Full backup, a Differential, which includes the first Incremental
134 backup, then two Incremental backups. All the above jobs would be written to
135 the {\bf Default} pool.
137 To consolidate those backups into a new Full backup, you would run the
141 run job=MyBackup level=VirtualFull
144 And it would produce a new Full backup without using the client, and the output
145 would be written to the {\bf Full} Pool which uses the Diskchanger Storage.
147 \section{Duplicate Job Control}
148 \index[general]{Duplicate Jobs}
149 The new version of Bacula provides four new directives that
150 give additional control over what Bacula does if duplicate jobs
151 are started. A duplicate job in the sense we use it here means
152 a second or subsequent job with the same name starts. This
153 happens most frequently when the first job runs longer than expected because no
156 The four directives each take as an argument a {\bf yes} or {\bf no} value and
157 are specified in the Job resource.
162 \item [Allow Duplicate Jobs = \lt{}yes|no\gt{}]
163 If this directive is enabled duplicate jobs will be run. If
164 the directive is set to {\bf no} (default) then only one job of a given name
165 may run at one time, and the action that Bacula takes to ensure only
166 one job runs is determined by the other directives (see below).
168 \item [Allow Higher Duplicates = \lt{}yes|no\gt{}]
169 If this directive is set to {\bf yes} (default) the job with a higher
170 priority (lower priority number) will be permitted to run. If the
171 priorities of the two jobs are the same, the outcome is determined by
172 other directives (see below).
174 \item [Cancel Queued Duplicates = \lt{}yes|no\gt{}]
175 If this directive is set to {\bf yes} (default) any job that is
176 already queued to run but not yet running will be canceled.
178 \item [Cancel Running Duplicates = \lt{}yes|no\gt{}]
179 If this directive is set to {\bf yes} any job that is already running
180 will be canceled. The default is {\bf no}.
183 \section{TLS Authentication}
184 \index[general]{TLS Authentication}
185 In Bacula version 2.5.x and later, in addition to the normal Bacula
186 CRAM-MD5 authentication that is used to authenticate each Bacula
187 connection, you can specify that you want TLS Authentication as well,
188 which will provide more secure authentication.
190 This new feature uses Bacula's existing TLS code (normally used for
191 communications encryption) to do authentication. To use it, you must
192 specify all the TLS directives normally used to enable communications
193 encryption (TLS Enable, TLS Verify Peer, TLS Certificate, ...) and
197 TLS Authenticate = yes
200 in the main daemon configuration resource (Director for the Director,
201 Client for the File daemon, and Storage for the Storage daemon).
203 When {\bf TLS Authenticate} is enabled, after doing the CRAM-MD5
204 authentication, Bacula will do the normal TLS authentication, then TLS
205 encryption will be turned off.
207 If you want to encrypt communications data, do not turn on {\bf TLS
210 \section{bextract non-portable Win32 data}
211 \index[general]{bextract handles Win32 non-portable data}
212 {\bf bextract} has been enhanced to be able to restore
213 non-portable Win32 data to any OS. Previous versions were
214 unable to restore non-portable Win32 data to machines that
215 did not have the Win32 BackupRead and BackupWrite API calls.
217 \section{State File updated at Job Termination}
218 \index[general]{State File}
219 In previous versions of Bacula, the state file, which provides a
220 summary of previous jobs run in the {\bf status} command output was
221 updated only when Bacula terminated, thus if the daemon crashed, the
222 state file might not contain all the run data. This version of
223 the Bacula daemons updates the state file on each job termination.
225 \section{MaxFullInterval}
226 \index[general]{MaxFullInterval}
227 The new Job resource directive {\bf Max Full Interval = \lt{}time-interval\gt{}}
228 can be used to specify the maximum time interval between {\bf Full} backup
229 jobs. When a job starts, if the time since the last Full backup is
230 greater than the specified interval, and the job would normally be an
231 {\bf Incremental} or {\bf Differential}, it will be automatically
232 upgraded to a {\bf Full} backup.
234 \section{MaxDiffInterval}
235 \index[general]{MaxDiffInterval}
236 The new Job resource directive {\bf Max Diff Interval = \lt{}time-interval\gt{}}
237 can be used to specify the maximum time interval between {\bf Differential} backup
238 jobs. When a job starts, if the time since the last Differential backup is
239 greater than the specified interval, and the job would normally be an
240 {\bf Incremental}, it will be automatically
241 upgraded to a {\bf Differential} backup.
243 \section{No Dump Flag}
244 \index[general]{MaxDiffInterval}
245 On FreeBSD systems, each file has a {\bf no dump flag} that can be set
246 by the user, and when it is set it is an indication to backup programs
247 to not backup that particular file. This version of Bacula contains a
248 new Options directive within a FileSet resource, which instructs Bacula to
249 obey this flag. The new directive is:
252 Honor No Dump Flag = yes|no
255 The default value is {\bf no}.
259 \index[general]{IgnoreDir}
260 The {\bf Ignore Dir = \lt{}filename\gt{}} is a new directive that can be added to the Include
261 section of the FileSet resource. If the specified
262 filename is found on the Client in any directory to be backed up,
263 the whole directory will be ignored (not backed up).
267 # List of files to be backed up
275 IgnoreDir = .excludeme
280 But in /home, there may be hundreds of directories of users and some
281 people want to indicate that they don't want to have certain
282 directories backed up. For example, with the above FileSet, if
283 the user or sysadmin creates a file named {\bf .excludeme} in
284 specific directories, such as
287 /home/user/www/cache/.excludeme
288 /home/user/temp/.excludeme
291 then Bacula will not backup the two directories named:
301 \section{Bacula Plugins}
302 \index[general]{Plugin}
303 Support for shared object plugins has been implemented in the Linux
304 (and Unix) File daemon. The API will be documented separately in
305 the Developer's Guide or in a new document. For the moment, there is
306 a single plugin named {\bf bpipe} that allows an external program to
307 get control to backup and restore a file.
309 Plugins are also planned (partially implemented) in the Director and the
310 Storage daemon. The code is also implemented to work on Win32 machines,
311 but it has not yet been tested.
313 \subsection{Plugin Directory}
314 Each daemon (DIR, FD, SD) has a new {\bf Plugin Directory} directive that may
315 be added to the daemon definition resource. The directory takes a quoted
316 string argument, which is the name of the directory in which the daemon can
317 find the Bacula plugins. If this directive is not specified, Bacula will not
318 load any plugins. Since each plugin has a distinctive name, all the daemons
319 can share the same plugin directory.
323 \subsection{Plugin Options}
324 The {\bf Plugin Options} directive takes a quoted string
325 arguement (after the equal sign) and may be specified in the
326 Job resource. The options specified will be passed to the plugin
327 when it is run. The value defined in the Job resource can be modified
328 by the user when he runs a Job via the {\bf bconsole} command line
331 Note: this directive may be specified, but it is not yet passed to
332 the plugin (i.e. not fully implemented).
334 \subsection{Plugin Options ACL}
335 The {\bf Plugin Options ACL} directive may be specified in the
336 Director's Console resource. It functions as all the other ACL commands
337 do by permitting users running restricted consoles to specify a
338 {\bf Plugin Options} that overrides the one specified in the Job
339 definition. Without this directive restricted consoles may not modify
343 The {\bf Plugin} directive is specified in the Include section of
344 a FileSet resource where you put your {\bf File = xxx} directives.
360 In the above example, when the File daemon is processing the directives
361 in the Include section, it will first backup all the files in {\bf /home}
362 then it will load the plugin named {\bf bpipe} (actually bpipe-dir.so) from
363 the Plugin Directory. The syntax and semantics of the Plugin directive
364 require the first part of the string up to the colon (:) to be the name
365 of the plugin. Everything after the first colon is ignored by the File daemon but
366 is passed to the plugin. Thus the plugin writer may define the meaning of the
367 rest of the string as he wishes.
369 Please see the next section for information about the {\bf bpipe} Bacula
372 \section{The bpipe Plugin}
373 The {\bf bpipe} plugin is provided in the directory src/plugins/fd/bpipe-fd.c of
374 the Bacula source distribution. When the plugin is compiled and linking into
375 the resulting dynamic shared object (DSO), it will have the name {\bf bpipe-fd.so}.
377 The purpose of the plugin is to provide an interface to any system program for
378 backup and restore. As specified above the {\bf bpipe} plugin is specified in
379 the Include section of your Job's FileSet resource. The full syntax of the
380 plugin directive as interpreted by the {\bf bpipe} plugin (each plugin is free
381 to specify the sytax as it wishes) is:
384 Plugin = "<field1>:<field2>:<field3>:<field4>"
389 \item {\bf field1} is the name of the plugin with the trailing {\bf -fd.so}
390 stripped off, so in this case, we would put {\bf bpipe} in this field.
392 \item {\bf field2} specifies the namespace, which for {\bf bpipe} is the
393 pseudo path and filename under which the backup will be saved. This pseudo
394 path and filename will be seen by the user in the restore file tree.
395 For example, if the value is {\bf /MYSQL/regress.sql}, the data
396 backed up by the plugin will be put under that "pseudo" path and filename.
397 You must be careful to choose a naming convention that is unique to avoid
398 a conflict with a path and filename that actually exists on your system.
400 \item {\bf field3} for the {\bf bpipe} plugin
401 specifies the "reader" program that is called by the plugin during
402 backup to read the data. {\bf bpipe} will call this program by doing a
405 \item {\bf field4} for the {\bf bpipe} plugin
406 specifies the "writer" program that is called by the plugin during
407 restore to write the data back to the filesystem.
410 Putting it all together, the full plugin directive line might look
414 Plugin = "bpipe:/MYSQL/regress.sql:mysqldump -f
415 --opt --databases bacula:mysql"
418 The directive has been split into two lines, but within the {\bf bacula-dir.conf} file
419 would be written on a single line.
421 This causes the File daemon to call the {\bf bpipe} plugin, which will write
422 its data into the "pseudo" file {\bf /MYSQL/regress.sql} by calling the
423 program {\bf mysqldump -f --opt --database bacula} to read the data during
424 backup. The mysqldump command outputs all the data for the database named
425 {\bf bacula}, which will be read by the plugin and stored in the backup.
426 During restore, the data that was backed up will be sent to the program
427 specified in the last field, which in this case is {\bf mysql}. When
428 {\bf mysql} is called, it will read the data sent to it by the plugn
429 then write it back to the same database from which it came ({\bf bacula}
432 The {\bf bpipe} plugin is a generic pipe program, that simply transmits
433 the data from a specified program to Bacula for backup, and then from Bacula to
434 a specified program for restore.
436 By using different command lines to {\bf bpipe},
437 you can backup any kind of data (ASCII or binary) depending
438 on the program called.
440 \section{libdbi framework}
441 As a general guideline, Bacula has support for a few catalog database drivers
442 coded natively by the Bacula team. With the libdbi implementation, which is a
443 Bacula driver that uses libdbi to access the catalog, we have an open field to
444 use many different kinds database engines following the needs of users.
446 The according to libdbi (http://libdbi.sourceforge.net/) project: libdbi
447 implements a database-independent abstraction layer in C, similar to the
448 DBI/DBD layer in Perl. Writing one generic set of code, programmers can
449 leverage the power of multiple databases and multiple simultaneous database
450 connections by using this framework.
452 Currently the libdbi driver in Bacula project only supports the same drivers
453 natively coded in Bacula. However the libdbi project has support for many
454 others database engines. You can view the list at
455 http://libdbi-drivers.sourceforge.net/. In the future all those drivers can be
456 supported by Bacula, however, they must be tested properly by the Bacula team.
458 Some of benefits of using libdbi are:
460 \item The possibility to use proprietary databases engines in which your
461 proprietary licenses prevent the Bacula team from developing the driver.
462 \item The possibility to use the drivers written for the libdbi project.
463 \item The possibility to use other database engines without recompiling Bacula
464 to use them. Just change one line in bacula-dir.conf
465 \item Abstract Database access, this is, unique point to code and profiling
466 catalog database access.
469 The following drivers have been tested:
471 \item PostgreSQL, with and without batch insert
472 \item Mysql, with and without batch insert
477 In the future, we will test and approve to use others databases engines
478 (proprietary or not) like DB2, Oracle, Microsoft SQL.
480 To compile Bacula to support libdbi we need to configure the code with the
481 --with-dbi and --with-dbi-driver=[database] ./configure options, where
482 [database] is the database engine to be used with Bacula (of course we can
483 change the driver in file bacula-dir.conf, see below). We must configure the
484 access port of the database engine with the option --with-db-port, because the
485 libdbi framework doesn't know the default access port of each database.
487 The next phase is checking (or configuring) the bacula-dir.conf, example:
491 dbdriver = dbi:mysql; dbaddress = 127.0.0.1; dbport = 3306
492 dbname = regress; user = regress; password = ""
496 The parameter {\bf dbdriver} indicates that we will use the driver dbi with a
497 mysql database. Currently the drivers supported by Bacula are: postgresql,
498 mysql, sqlite, sqlite3; these are the names that may be added to string "dbi:".
500 The following limitations apply when Bacula is set to use the libdbi framework:
501 - Not tested on the Win32 platform
502 - A little performance is lost if comparing with native database driver.
503 The reason is bound with the database driver provided by libdbi and the
504 simple fact that one more layer of code was added.
506 It is important to remember, when compiling Bacula with libdbi, the
507 following packages are needed:
509 \item libdbi version 1.0.0, http://libdbi.sourceforge.net/
510 \item libdbi-drivers 1.0.0, http://libdbi-drivers.sourceforge.net/
513 You can download them and compile them on your system or install the packages
514 from your OS distribution.
517 \section{Display Autochanger Content}
518 \index[general]{StatusSlots}
520 The {\bf status slots storage=\lt{}storage-name\gt{}} command displays autochanger content.
524 Slot | Volume Name | Status | Media Type | Pool | Loaded |
525 ------+---------------+----------+-------------------+------------+---------|
526 1 | 00001 | Append | DiskChangerMedia | Default | 0 |
527 2 | 00002 | Append | DiskChangerMedia | Default | 0 |
528 3*| 00003 | Append | DiskChangerMedia | Scratch | 0 |
533 If you an asterisk ({\bf *}) appears after the slot number, you must run an
534 {\bf update slots} command to synchronize autochanger content with your
537 \section{Miscellaneous}
538 \index[general]{Misc New Features}
540 \item [Virtual Tape Emulation]
542 We now have a Virtual Tape emulator that allows us to run though 99.9\% of
543 the tape code but actually reading and writing to a disk file. Used with the
544 \textbf{disk-changer} script, you can now emulate an autochanger with 10 drives
545 and 700 slots. This feature is most useful in testing. It is enabled
546 by using {\bf Device Type = vtape} in the Storage daemon's Device
547 directive. This feature is only implemented on Linux machines.
549 \item [Bat Enhancements]
550 Bat (the Bacula Administration Tool) GUI program has been significantly
551 enhanced and stabilized. In particular, there are new table based status
552 commands; it can now be easily localized using Qt4 Linguist.
554 The Bat communications protocol has been significantly enhanced to improve
557 \item [RunScript Enhancements]
558 The {\bf RunScript} resource has been enhanced to permit multiple
559 commands per RunScript. Simply specify multiple {\bf Command} directives
566 Command = "/bin/echo test"
567 Command = "/bin/echo an other test"
568 Command = "/bin/echo 3 commands in the same runscript"
575 A new Client RunScript {\bf RunsWhen} keyword of {\bf AfterVSS} has been implemented, which
576 runs the command after the Volume Shadow Copy has been made.
578 Console commands can be specified within a RunScript by using:
579 {\bf Console = \lt{}command\gt{}}, however, this command has not been
580 carefully tested and debugged and is known to easily crash the Director.
581 We would appreciate feedback. Due to the recursive nature of this command, we
582 may remove it before the final release.
584 \item [Status Enhancements]
585 The bconsole {\bf status dir} output has been enhanced to indicate
586 Storage daemon job spooling and despooling activity.
588 \item [Connect Timeout]
589 The default connect timeout to the File
590 daemon has been set to 3 minutes. Previously it was 30 minutes.
592 \item [ftruncate for NFS Volumes]
593 If you write to a Volume mounted by NFS (say on a local file server),
594 in previous Bacula versions, when the Volume was recycled, it was not
595 properly truncated because NFS does not implement ftruncate (file
596 truncate). This is now corrected in the new version because we have
597 written code (actually a kind user) that deletes and recreates the Volume,
598 thus accomplishing the same thing as a truncate.
600 \item [Support for Ubuntu]
601 The new version of Bacula now recognizes the Ubuntu (and Kubuntu)
602 version of Linux, and thus now provides correct autostart routines.
603 Since Ubuntu officially supports Bacula, you can also obtain any
604 recent release of Bacula from the Ubuntu repositories.
606 \item[Recycling enhancements]
607 The new \textbf{RecyclePool} directive defines to which pool the Volume will
608 be placed (moved) when it is recycled. Without this directive, a Volume will
609 remain in the same pool when it is recycled. With this directive, it can be
610 moved automatically to any existing pool during a recycle. This directive is
611 probably most useful when defined in the Scratch pool, so that volumes will
612 be recycled back into the Scratch pool.
615 The File daemon to Director protocol now includes a version
616 number, which will help us in future versions automatically determine
617 if a File daemon is not compatible.
619 \item [Max Run Sched Time]
620 The time specifies the maximum allowed time that a job may run, counted from
621 when the job was scheduled. This can be useful to prevent jobs from running
622 during working hours. We can see it like \texttt{Max Start Delay + Max Run
625 \item [Max Wait Time]
626 Previous \textbf{MaxWaitTime} directives aren't working as expected, instead
627 of checking the maximum allowed time that a job may block for a resource,
628 those directives worked like \textbf{MaxRunTime}. Some users are reporting to use
629 \textbf{Incr/Diff/Full Max Wait Time} to control the maximum run time of their job
630 depending on the level. Now, they have to use \textbf{Incr/Diff/Full Max Run Time}.
631 \textbf{Incr/Diff/Full Max Wait Time} directives are now deprecated.
633 \item [Incremental|Differential Max Wait Time = \lt{}time\gt{}]
634 Theses directives have been deprecated in favor of \texttt{Incremental|Differential
637 \item [Max Run Time directives]
638 Using \textbf{Full/Diff/Incr Max Run Time}, it's now possible to specify the
639 maximum allowed time that a job can run depending on the level.
641 \addcontentsline{lof}{figure}{Job time control directives}
642 \includegraphics{\idir different_time.eps}
644 \item [Statistics Enhancements]
645 If you (or your boss) want to have statistics on your backups, you could use
646 a few SQL queries on the Job table to report how many:
649 \item jobs have been successful
650 \item files have been backed up
654 However, these statistics are accurate only if your job retention is greater than
655 your statistics period. Ie, if jobs are purged from the catalog, you won't be
658 Now, you can use the \textbf{update stats [days=num]} console command to fill the
659 JobStat table with new Job records.
661 The \textbf{Statistics Retention = \lt{}time\gt{}} director directive defines
662 the length of time that Bacula will keep statistics job records in the Catalog
663 database after the Job End time. (In \texttt{JobStat} table) When this time
664 period expires, and if user runs \texttt{prune stats} command, Bacula will
665 prune (remove) Job records that are older than the specified period.
667 These statistics records aren't used for restore purpose, but mainly for
668 capacity planning, billings, etc.
670 You can use the following Job resource in your nightly \textbf{BackupCatalog}
671 job to maintain statistics.
677 Console = "update stats days=3"
678 Console = "prune stats yes"
685 \item [Spooling Enhancements]
686 A new job directive permits to specify the spool size per job. This is used
687 in advanced job tunning. {\bf SpoolSize={\it bytes}}
691 \section{Building Bacula Plugins}
692 There is currently one sample program {\bf example-plugin-fd.c} and
693 one working plugin {\bf bpipe-fd.c} that can be found in the Bacula
694 {\bf src/plugins/fd} directory. Both are built with the following:
698 ./configure <your-options>
706 After building Bacula and changing into the src/plugins/fd directory,
707 the {\bf make} command will build the {\bf bpipe-fd.so} plugin, which
708 is a very useful and working program.
710 The {\bf make test} command will build the {\bf example-plugin-fd.so}
711 plugin and a binary named {\bf main}, which is build from the source
712 code located in {\bf src/filed/fd\_plugins.c}.
714 If you execute {\bf ./main}, it will load and run the example-plugin-fd
715 plugin simulating a small number of the calling sequences that Bacula uses
716 in calling a real plugin. This allows you to do initial testing of
717 your plugin prior to trying it with Bacula.
719 You can get a good idea of how to write your own plugin by first
720 studying the example-plugin-fd, and actually running it. Then
721 it can also be instructive to read the bpipe-fd.c code as it is
722 a real plugin, which is still rather simple and small.
724 When actually writing your own plugin, you may use the example-plugin-fd.c
725 code as a template for your code.
731 \chapter{Bacula FD Plugin API}
732 To write a Bacula plugin, you create a dynamic shared object
733 program (or dll on Win32) with a particular name and two
734 exported entry points, place it in the {\bf Plugins Directory}, which is defined in the
735 {\bf bacula-fd.conf} file in the {\bf Client} resource, and when the FD
736 starts, it will load all the plugins that end with {\bf -fd.so} (or {\bf -fd.dll}
737 on Win32) found in that directory.
739 Once the File daemon loads the plugins, it asks the OS for the
740 two entry points (loadPlugin and unloadPlugin) then calls the
741 {\bf loadPlugin} entry point (see below).
743 Bacula passes information to the plugin through this call and it gets
744 back information that it needs to use the plugin. Later, Bacula
745 will call particular functions that are defined by the
746 {\bf loadPlugin} interface.
748 When Bacula is finished with the plugin
749 (when Bacula is going to exit), it will call the {\bf unloadPlugin}
752 The two entry points are:
755 bRC loadPlugin(bInfo *lbinfo, bFuncs *lbfuncs, pInfo **pinfo, pFuncs **pfuncs)
762 both these external entry points to the shared object are defined as C entry points
763 to avoid name mangling complications with C++. However, the shared object
764 can actually be written in any language (preferrably C or C++) providing that it
765 follows C language calling conventions.
767 The definitions for {\bf bRC} and the arguments are {\bf
768 src/filed/fd-plugins.h} and so this header file needs to be included in
769 your plugin. It along with {\bf src/lib/plugins.h} define basically the whole
770 plugin interface. Within this header file, it includes the following
774 #include <sys/types.h>
776 #include "bc_types.h"
777 #include "lib/plugins.h"
778 #include <sys/stat.h>
781 Aside from the {\bf bc\_types.h} and {\bf confit.h} headers, the plugin definition uses the
782 minimum code from Bacula. The bc\_types.h file is required to ensure that
783 the data type defintions in arguments correspond to the Bacula core code.
785 The return codes are defined as:
789 bRC_Stop = 1, /* Stop calling other plugins */
790 bRC_Error = 2, /* Some kind of error */
791 bRC_More = 3, /* More files to backup */
796 At a future point in time, we hope to make the Bacula libbac.a into a
797 shared object so that the plugin can use much more of Bacula's
798 infrastructure, but for this first cut, we have tried to minimize the
799 dependence on Bacula.
802 As previously mentioned, the {\bf loadPlugin} entry point in the plugin
803 is called immediately after Bacula loads the plugin when the File daemon
804 itself is first starting. This entry point is only called once during the
805 execution of the File daemon. In calling the
806 plugin, the first two arguments are information from Bacula that
807 is passed to the plugin, and the last two arguments are information
808 about the plugin that the plugin must return to Bacula. The call is:
811 bRC loadPlugin(bInfo *lbinfo, bFuncs *lbfuncs, pInfo **pinfo, pFuncs **pfuncs)
814 and the arguments are:
818 This is information about Bacula in general. Currently, the only value
819 defined in the bInfo structure is the version, which is the Bacula plugin
820 interface version, currently defined as 1. The {\bf size} is set to the
821 byte size of the structure. The exact definition of the bInfo structure
822 as of this writing is:
825 typedef struct s_baculaInfo {
832 The bFuncs structure defines the callback entry points within Bacula
833 that the plugin can use register events, get Bacula values, set
834 Bacula values, and send messages to the Job output or debug output.
836 The exact definition as of this writing is:
838 typedef struct s_baculaFuncs {
841 bRC (*registerBaculaEvents)(bpContext *ctx, ...);
842 bRC (*getBaculaValue)(bpContext *ctx, bVariable var, void *value);
843 bRC (*setBaculaValue)(bpContext *ctx, bVariable var, void *value);
844 bRC (*JobMessage)(bpContext *ctx, const char *file, int line,
845 int type, time_t mtime, const char *fmt, ...);
846 bRC (*DebugMessage)(bpContext *ctx, const char *file, int line,
847 int level, const char *fmt, ...);
851 We will discuss these entry points and how to use them a bit later when
852 describing the plugin code.
856 When the loadPlugin entry point is called, the plugin must initialize
857 an information structure about the plugin and return a pointer to
858 this structure to Bacula.
860 The exact definition as of this writing is:
863 typedef struct s_pluginInfo {
866 const char *plugin_magic;
867 const char *plugin_license;
868 const char *plugin_author;
869 const char *plugin_date;
870 const char *plugin_version;
871 const char *plugin_description;
877 \item [version] is the current Bacula defined plugin interface version, currently
878 set to 1. If the interface version differs from the current version of
879 Bacula, the plugin will not be run (not yet implemented).
880 \item [plugin\_magic] is a pointer to the text string "*FDPluginData*", a
881 sort of sanity check. If this value is not specified, the plugin
882 will not be run (not yet implemented).
883 \item [plugin\_license] is a pointer to a text string that describes the
884 plugin license. Bacula will only accept compatible licenses (not yet
886 \item [plugin\_author] is a pointer to the text name of the author of the program.
887 This string can be anything but is generally the author's name.
888 \item [plugin\_date] is the pointer text string containing the date of the plugin.
889 This string can be anything but is generally some human readable form of
891 \item [plugin\_version] is a pointer to a text string containing the version of
892 the plugin. The contents are determined by the plugin writer.
893 \item [plugin\_description] is a pointer to a string describing what the
894 plugin does. The contents are determined by the plugin writer.
897 The pInfo structure must be defined in static memory because Bacula does not
898 copy it and may refer to the values at any time while the plugin is
899 loaded. All values must be supplied or the plugin will not run (not yet
900 implemented). All text strings must be either ASCII or UTF-8 strings that
901 are terminated with a zero byte.
904 When the loadPlugin entry point is called, the plugin must initialize
905 an entry point structure about the plugin and return a pointer to
906 this structure to Bacula. This structure contains pointer to each
907 of the entry points that the plugin must provide for Bacula. When
908 Bacula is actually running the plugin, it will call the defined
909 entry points at particular times. All entry points must be defined.
911 The pFuncs structure must be defined in static memory because Bacula does not
912 copy it and may refer to the values at any time while the plugin is
915 The exact definition as of this writing is:
918 typedef struct s_pluginFuncs {
921 bRC (*newPlugin)(bpContext *ctx);
922 bRC (*freePlugin)(bpContext *ctx);
923 bRC (*getPluginValue)(bpContext *ctx, pVariable var, void *value);
924 bRC (*setPluginValue)(bpContext *ctx, pVariable var, void *value);
925 bRC (*handlePluginEvent)(bpContext *ctx, bEvent *event, void *value);
926 bRC (*startBackupFile)(bpContext *ctx, struct save_pkt *sp);
927 bRC (*endBackupFile)(bpContext *ctx);
928 bRC (*startRestoreFile)(bpContext *ctx, const char *cmd);
929 bRC (*endRestoreFile)(bpContext *ctx);
930 bRC (*pluginIO)(bpContext *ctx, struct io_pkt *io);
931 bRC (*createFile)(bpContext *ctx, struct restore_pkt *rp);
932 bRC (*setFileAttributes)(bpContext *ctx, struct restore_pkt *rp);
936 The details of the entry points will be presented in
937 separate sections below.
941 \item [size] is the byte size of the structure.
942 \item [version] is the plugin interface version currently set to 1.
945 Sample code for loadPlugin:
947 bfuncs = lbfuncs; /* set Bacula funct pointers */
949 *pinfo = &pluginInfo; /* return pointer to our info */
950 *pfuncs = &pluginFuncs; /* return pointer to our functions */
955 where pluginInfo and pluginFuncs are statically defined structures.
956 See bpipe-fd.c for details.
962 \section{Plugin Entry Points}
963 This section will describe each of the entry points (subroutines) within
964 the plugin that the plugin must provide for Bacula, when they are called
965 and their arguments. As noted above, pointers to these subroutines are
966 passed back to Bacula in the pFuncs structure when Bacula calls the
967 loadPlugin() externally defined entry point.
969 \subsection{newPlugin(bpContext *ctx)}
970 This is the entry point that Bacula will call
971 when a new "instance" of the plugin is created. This typically
972 happens at the beginning of a Job. If 10 Jobs are running
973 simultaneously, there will be at least 10 instances of the
976 The bpContext structure will be passed to the plugin, and
977 during this call, if the plugin needs to have its private
978 working storage that is associated with the particular
979 instance of the plugin, it should create it from the heap
980 (malloc the memory) and store a pointer to
981 its private working storage in the {\bf pContext} variable.
982 Note: since Bacula is a multi-threaded program, you must not
983 keep any variable data in your plugin unless it is truely meant
984 to apply globally to the whole plugin. In addition, you must
985 be aware that except the first and last call to the plugin
986 (loadPlugin and unloadPlugin) all the other calls will be
987 made by threads that correspond to a Bacula job. The
988 bpContext that will be passed for each thread will remain the
989 same throughout the Job thus you can keep your privat Job specific
990 data in it ({\bf bContext}).
993 typedef struct s_bpContext {
994 void *pContext; /* Plugin private context */
995 void *bContext; /* Bacula private context */
1000 This context pointer will be passed as the first argument to all
1001 the entry points that Bacula calls within the plugin. Needless
1002 to say, the plugin should not change the bContext variable, which
1003 is Bacula's private context pointer for this instance (Job) of this
1006 \subsection{freePlugin(bpContext *ctx)}
1007 This entry point is called when the
1008 this instance of the plugin is no longer needed (the Job is
1009 ending), and the plugin should release all memory it may
1010 have allocated for this particular instance (Job) i.e. the pContext.
1011 This is not the final termination
1012 of the plugin signaled by a call to {\bf unloadPlugin}.
1013 Any other instances (Job) will
1014 continue to run, and the entry point {\bf newPlugin} may be called
1015 again if other jobs start.
1017 \subsection{getPluginValue(bpContext *ctx, pVariable var, void *value)}
1018 Bacula will call this entry point to get
1019 a value from the plugin. This entry point is currently not called.
1021 \subsection{setPluginValue(bpContext *ctx, pVariable var, void *value)}
1022 Bacula will call this entry point to set
1023 a value in the plugin. This entry point is currently not called.
1025 \subsection{handlePluginEvent(bpContext *ctx, bEvent *event, void *value)}
1026 This entry point is called when Bacula
1027 encounters certain events (discussed below). This is, in fact, the
1028 main way that most plugins get control when a Job runs and how
1029 they know what is happening in the job. It can be likened to the
1030 {\bf RunScript} feature that calls external programs and scripts,
1031 and is very similar to the Bacula Python interface.
1032 When the plugin is called, Bacula passes it the pointer to an event
1033 structure (bEvent), which currently has one item, the eventType:
1036 typedef struct s_bEvent {
1041 which defines what event has been triggered, and for each event,
1042 Bacula will pass a pointer to a value associated with that event.
1043 If no value is associated with a particular event, Bacula will
1044 pass a NULL pointer, so the plugin must be careful to always check
1045 value pointer prior to dereferencing it.
1047 The current list of events are:
1053 bEventStartBackupJob = 3,
1054 bEventEndBackupJob = 4,
1055 bEventStartRestoreJob = 5,
1056 bEventEndRestoreJob = 6,
1057 bEventStartVerifyJob = 7,
1058 bEventEndVerifyJob = 8,
1059 bEventBackupCommand = 9,
1060 bEventRestoreCommand = 10,
1067 Most of the above are self-explanatory.
1070 \item [bEventJobStart] is called whenever a Job starts. The value
1071 passed is a pointer to a string that contains: "Jobid=nnn
1072 Job=job-name". Where nnn will be replaced by the JobId and job-name
1073 will be replaced by the Job name. The variable is temporary so if you
1074 need the values, you must copy them.
1076 \item [bEventJobEnd] is called whenever a Job ends. No value is passed.
1078 \item [bEventStartBackupJob] is called when a Backup Job begins. No value
1081 \item [bEventEndBackupJob] is called when a Backup Job ends. No value is
1084 \item [bEventStartRestoreJob] is called when a Restore Job starts. No value
1087 \item [bEventEndRestoreJob] is called when a Restore Job ends. No value is
1090 \item [bEventStartVerifyJob] is called when a Verify Job starts. No value
1093 \item [bEventEndVerifyJob] is called when a Verify Job ends. No value
1096 \item [bEventBackupCommand] is called prior to the bEventStartBackupJob and
1097 the plugin is passed the command string (everything after the equal sign
1098 in "Plugin =" as the value.
1100 Note, if you intend to backup a file, this is an important first point to
1101 write code that copies the command string passed into your pContext area
1102 so that you will know that a backup is being performed and you will know
1103 the full contents of the "Plugin =" command (i.e. what to backup and
1104 what virtual filename the user wants to call it.
1106 \item [bEventRestoreCommand] is called prior to the bEventStartRestoreJob and
1107 the plugin is passed the command string (everything after the equal sign
1108 in "Plugin =" as the value.
1110 See the notes above concerning backup and the command string. This is the
1111 point at which Bacula passes you the original command string that was
1112 specified during the backup, so you will want to save it in your pContext
1113 area for later use when Bacula calls the plugin again.
1115 \item [bEventLevel] is called when the level is set for a new Job. The value
1116 is a 32 bit integer stored in the void*, which represents the Job Level code.
1118 \item [bEventSince] is called when the since time is set for a new Job. The
1119 value is a time\_t time at which the last job was run.
1122 During each of the above calls, the plugin receives either no specific value or
1123 only one value, which in some cases may not be sufficient. However, knowing the
1124 context of the event, the plugin can call back to the Bacula entry points it
1125 was passed during the {\bf loadPlugin} call and get to a number of Bacula variables.
1126 (at the current time few Bacula variables are implemented, but it easily extended
1127 at a future time and as needs require).
1129 \subsection{startBackupFile(bpContext *ctx, struct save\_pkt *sp)}
1130 Called when beginning the backup of a file. Here Bacula provides you
1131 with a pointer to the {\bf save\_pkt} structure and you must fill in
1132 this packet with the "attribute" data of the file.
1136 char *fname; /* Full path and filename */
1137 char *link; /* Link name if any */
1138 struct stat statp; /* System stat() packet for file */
1139 int32_t type; /* FT_xx for this file */
1140 uint32_t flags; /* Bacula internal flags */
1141 bool portable; /* set if data format is portable */
1142 char *cmd; /* command */
1146 The second argument is a pointer to the {\bf save\_pkt} structure for the file
1147 to be backed up. The plugin is responsible for filling in all the fields
1148 of the {\bf save\_pkt}. If you are backing up
1149 a real file, then generally, the statp structure can be filled in by doing
1150 a {\bf stat} system call on the file.
1152 If you are backing up a database or
1153 something that is more complex, you might want to create a virtual file.
1154 That is a file that does not actually exist on the filesystem, but represents
1155 say an object that you are backing up. In that case, you need to ensure
1156 that the {\bf fname} string that you pass back is unique so that it
1157 does not conflict with a real file on the system, and you need to
1158 artifically create values in the statp packet.
1160 Example programs such as {\bf bpipe-fd.c} show how to set these fields.
1161 You must take care not to store pointers the stack in the pointer fields such
1162 as fname and link, because when you return from your function, your stack entries
1163 will be destroyed. The solution in that case is to malloc() and return the pointer
1164 to it. In order to not have memory leaks, you should store a pointer to all memory
1165 allocated in your pContext structure so that in subsequent calls or at termination,
1166 you can release it back to the system.
1168 Once the backup has begun, Bacula will call your plugin at the {\bf pluginIO}
1169 entry point to "read" the data to be backed up. Please see the {\bf bpipe-fd.c}
1170 plugin for how to do I/O.
1172 Example of filling in the save\_pkt as used in bpipe-fd.c:
1175 struct plugin_ctx *p_ctx = (struct plugin_ctx *)ctx->pContext;
1176 time_t now = time(NULL);
1177 sp->fname = p_ctx->fname;
1178 sp->statp.st_mode = 0700 | S_IFREG;
1179 sp->statp.st_ctime = now;
1180 sp->statp.st_mtime = now;
1181 sp->statp.st_atime = now;
1182 sp->statp.st_size = -1;
1183 sp->statp.st_blksize = 4096;
1184 sp->statp.st_blocks = 1;
1185 p_ctx->backup = true;
1189 Note: the filename to be created has already been created from the
1190 command string previously sent to the plugin and is in the plugin
1191 context (p\_ctx->fname) and is a malloc()ed string. This example
1192 creates a regular file (S\_IFREG), with various fields being created.
1194 In general, the sequence of commands issued from Bacula to the plugin
1195 to do a backup while processing the "Plugin = " directive are:
1198 \item generate a bEventBackupCommand event to the specified plugin
1199 and pass it the command string.
1200 \item make a startPluginBackup call to the plugin, which
1201 fills in the data needed in save\_pkt to save as the file
1202 attributes and to put on the Volume and in the catalog.
1203 \item call Bacula's internal save\_file() subroutine to save the specified
1204 file. The plugin will then be called at pluginIO() to "open"
1205 the file, and then to read the file data.
1206 Note, if you are dealing with a virtual file, the "open" operation
1207 is something the plugin does internally and it doesn't necessarily
1208 mean opening a file on the filesystem. For example in the case of
1209 the bpipe-fd.c program, it initiates a pipe to the requested program.
1210 Finally when the plugin signals to Bacula that all the data was read,
1211 Bacula will call the plugin with the "close" pluginIO() function.
1215 \subsection{endBackupFile(bpContext *ctx)}
1216 Called at the end of backing up a file. If the plugin's work
1217 is done, it should return bRC\_OK. If the plugin wishes to create another
1218 file and back it up, then it must return bRC\_More (not yet implemented).
1219 This is probably a good time to release any malloc()ed memory you used to
1220 pass back filenames.
1222 \subsection{startRestoreFile(bpContext *ctx, const char *cmd)}
1226 \subsection{endRestoreFile(bpContext *ctx)}
1227 Called when done restoring a file.
1229 \subsection{pluginIO(bpContext *ctx, struct io\_pkt *io)}
1230 Called to do the input (backup) or output (restore) of data from or to a
1243 int32_t func; /* Function code */
1244 int32_t count; /* read/write count */
1245 mode_t mode; /* permissions for created files */
1246 int32_t flags; /* open flags (e.g. O_WRONLY ...) */
1247 char *buf; /* read/write buffer */
1248 int32_t status; /* return status */
1249 int32_t io_errno; /* errno code */
1256 \subsection{createFile(bpContext *ctx, struct restore\_pkt *rp)}
1257 Called to create a file during a Restore job before restoring the data. The data in the
1258 restore\_pkt is passed to the plugin and is based on the data that was
1259 originally given by the plugin during the backup and the current user
1260 restore settings (e.g. where, RegexWhere, replace). This allows the
1261 plugin to first create a file (if necessary) so that the data can
1262 be transmitted to it. The next call to the plugin will be a
1263 pluginIO command with a request to open the file write-only.
1267 struct restore_pkt {
1268 int32_t stream; /* attribute stream id */
1269 int32_t data_stream; /* id of data stream to follow */
1270 int32_t type; /* file type FT */
1271 int32_t file_index; /* file index */
1272 int32_t LinkFI; /* file index to data if hard link */
1273 uid_t uid; /* userid */
1274 struct stat statp; /* decoded stat packet */
1275 const char *attrEx; /* extended attributes if any */
1276 const char *ofname; /* output filename */
1277 const char *olname; /* output link name */
1278 const char *where; /* where */
1279 const char *RegexWhere; /* regex where */
1280 int replace; /* replace flag */
1284 \subsection{setFileAttributes(bpContext *ctx, struct restore\_pkt *rp)}
1285 This is call not yet implemented.
1288 struct restore_pkt {
1289 int32_t stream; /* attribute stream id */
1290 int32_t data_stream; /* id of data stream to follow */
1291 int32_t type; /* file type FT */
1292 int32_t file_index; /* file index */
1293 int32_t LinkFI; /* file index to data if hard link */
1294 uid_t uid; /* userid */
1295 struct stat statp; /* decoded stat packet */
1296 const char *attrEx; /* extended attributes if any */
1297 const char *ofname; /* output filename */
1298 const char *olname; /* output link name */
1299 const char *where; /* where */
1300 const char *RegexWhere; /* regex where */
1301 int replace; /* replace flag */