5 Frontend, backend, database, callback, overlay - what does it all mean?
7 The "frontend" refers to all the code that deals with the actual interaction
8 with an LDAP client. This includes the code to read requests from the network
9 and parse them into C data structures, all of the session management, and the
10 formatting of responses for transmission onto the network. It also includes the
11 access control engine and other features that are generic to LDAP processing,
12 features which are not dependent on a particular database implementation.
13 Because the frontend serves as the framework that ties everything together,
14 it should not change much over time.
16 The terms "backend" and "database" have historically been used interchangeably
17 and/or in combination as if they are the same thing, but the code has a clear
18 distinction between the two. A "backend" is a type of module, and a "database"
19 is an instance of a backend type. Together they work with the frontend to
20 manage the actual data that are operated on by LDAP requests. Originally the
21 backend interface was relatively compact, with individual functions
22 corresponding to each LDAP operation type, plus functions for init, config, and
23 shutdown. The number of entry points has grown to allow greater flexibility,
24 but the concept is much the same as before.
26 The language here can get a bit confusing. A backend in slapd is embodied in a
27 BackendInfo data structure, and a database is held in a BackendDB structure.
28 Originally it was all just a single Backend data structure, but things have
29 grown and the concept was split into these two parts. The idea behind the
30 distinct BackendInfo is to allow for any initialization and configuration that
31 may be needed by every instance of a type of database, as opposed to items that
32 are specific to just one instance. For example, you might have a database
33 library that requires an initialization routine to be called exactly once at
34 program startup. Then there may be a "open" function that must be called once
35 for each database instance. The BackendInfo.bi_open function provides the
36 one-time startup, while the BackendInfo.bi_db_open function provides the
37 per-database startup. The main feature of the BackendInfo structure is its
38 table of entry points for all of the database functions that it implements.
39 There's also a bi_private pointer that can be used to carry any configuration
40 state needed by the backend. (Note that this is state that applies to the
41 backend type, and thus to all database instances of the backend as well.) The
42 BackendDB structure carries all of the per-instance state for a backend
43 database. This includes the database suffix, ACLs, flags, various DNs, etc. It
44 also has a pointer to its BackendInfo, and a be_private pointer for use by the
45 particular backend instance. In practice, the per-type features are seldom
46 used, and all of the work is done in the per-instance data structures.
48 Ordinarily an LDAP request is received by the slapd frontend, parsed into a
49 request structure, and then passed to the backend for processing. The backend
50 may call various utility functions in the frontend to assist in processing, and
51 then it eventually calls some send_ldap_result function in the frontend to send
52 results back to the client. The processing flow is pretty rigidly defined; even
53 though slapd is capable of dynamically loading new code modules, it was
54 difficult to add extensions that changed the basic protocol operations. If you
55 wanted to extend the server with special behaviors you would need to modify the
56 frontend or the backend or both, and generally you would need to write an
57 entire new backend to get some set of special features working. With OpenLDAP
58 2.1 we added the notion of a callback, which can intercept the results sent
59 from a backend before they are sent to a client. Using callbacks makes it
60 possible to modify the results if desired, or to simply discard the results
61 instead of sending them to any client. This callback feature is used
62 extensively in the SASL support to perform internal searches of slapd databases
63 when mapping authentication IDs into regular DNs. The callback mechanism is
64 also the basis of backglue, which allows separate databases to be searched as
65 if they were a single naming context.
67 Very often, one needs to add just a tiny feature onto an otherwise "normal"
68 database. The usual way to achieve this was to use a programmable backend (like
69 back-perl) to preprocess various requests and then forward them back into slapd
70 to be handled by the real database. While this technique works, it is fairly
71 inefficient because it involves many transitions from network to slapd and back
72 again. The overlay concept introduced in OpenLDAP 2.2 allows code to be
73 inserted between the slapd frontend and any backend, so that incoming requests
74 can be intercepted before reaching the backend database. (There is also a SLAPI
75 plugin framework in OpenLDAP 2.2; it offers a lot of flexibility as well but is
76 not discussed here.) The overlay framework also uses the callback mechanism, so
77 outgoing results can also be intercepted by external code. All of this could
78 get unwieldy if a lot of overlays were being used, but there was also another
79 significant API change in OpenLDAP 2.2 to streamline internal processing. (See
80 the document "Refactoring the slapd ABI"...)
82 OK, enough generalities... You should probably have a copy of slap.h in front
83 of you to continue here.
85 What is an overlay? The structure defining it includes a BackendInfo structure
86 plus a few additional fields. It gets inserted into the usual frontend->backend
87 call chain by replacing the BackendDB's BackendInfo pointer with its own. The
88 framework to accomplish this is in backover.c. For a given backend, the
89 BackendInfo will point to a slap_overinfo structure. The slap_overinfo has a
90 BackendInfo that points to all of the overlay framework's entry points. It also
91 holds a copy of the original BackendInfo pointer, and a linked list of
92 slap_overinst structures. There is one slap_overinst per configured overlay,
93 and the set of overlays configured on a backend are treated like a stack; i.e.,
94 the last one configured is at the top of the stack, and it executes first.
96 Continuing with the stack notion - a request enters the frontend, is directed
97 to a backend by select_backend, and then intercepted by the top of the overlay
98 stack. This first overlay may do something with the request, and then return
99 SLAP_CB_CONTINUE, which will then cause processing to fall into the next
100 overlay, and so on down the stack until finally the request is handed to the
101 actual backend database. Likewise, when the database finishes processing and
102 sends a result, the overlay callback intercepts this and the topmost overlay
103 gets to process the result. If it returns SLAP_CB_CONTINUE then processing will
104 continue in the next overlay, and then any other callbacks, then finally the
105 result reaches the frontend for sending back to the client. At any step along
106 the way, a module may choose to fully process the request or result and not
107 allow it to propagate any further down the stack. Whenever a module returns
108 anything other than SLAP_CB_CONTINUE the processing stops.
110 An overlay can call most frontend functions without any special consideration.
111 However, if a call is going to result in any backend code being invoked, then
112 the backend environment must be correct. During a normal backend invocation,
113 op->o_bd points to the BackendDB structure for the backend, and
114 op->o_bd->bd_info points to the BackendInfo for the backend. All of the
115 information a specific backend instance needs is in op->o_bd->be_private and
116 all of its entry points are in the BackendInfo structure. When overlays are in
117 use on a backend, op->o_bd->bd_info points to the BackendInfo (actually a
118 slap_overinfo) that contains the overlay framework. When a particular overlay
119 instance is executing, op->o_bd points to a copy of the original op->o_bd, and
120 op->o_bd->bd_info points to a slap_overinst which carries the information about
121 the current overlay. The slap_overinst contains an on_private pointer which can
122 be used to carry any configuration or state information the overlay needs. The
123 normal way to invoke a backend function is through the op->o_bd->bd_info table
124 of entry points, but obviously this must be set to the backend's original
125 BackendInfo in order to get to the right function.
127 There are two approaches here. The slap_overinst also contains a on_info field
128 that points to the top slap_overinfo that wraps the current backend. The
129 simplest thing is for the overlay to set op->o_bd->bd_info to this on_info
130 value before invoking a backend function. This will cause processing of that
131 particular operation to begin at the top of the overlay stack, so all the other
132 overlays on the backend will also get a chance to handle this internal request.
133 The other possibility is to invoke the underlying backend directly, bypassing
134 the rest of the overlays, by calling through on_info->oi_orig. You should be
135 careful in choosing this approach, since it precludes other overlays from doing
138 One of the more interesting uses for an overlay is to attach two (or more)
139 different database backends into a single execution stack. Assuming that the
140 basic frontend-managed information (suffix, rootdn, ACLs, etc.) will be the
141 same for all of the backends, the only thing the overlay needs to maintain is a
142 be_private and bd_info pointer for the added backends. The chain and proxycache
143 overlays are two complementary examples of this usage. The chain overlay
144 attaches a back-ldap backend to a local database backend, and allows referrals
145 to remote servers generated by the database to be processed by slapd instead of
146 being returned to the client. The proxycache overlay attaches a local database
147 to a back-ldap (or back-meta) backend and allows search results from remote
148 servers to be cached locally. In both cases the overlays must provide a bit of
149 glue to swap in the appropriate be_private and bd_info pointers before invoking
150 the attached backend, which can then be invoked as usual.
projects / openldap / blob