2 # Copyright 1999-2007 The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
7 Replicated directories are a fundamental requirement for delivering a
8 resilient enterprise deployment.
10 OpenLDAP has various configuration options for creating a replicated
11 directory. The following sections will discuss these.
13 H2: Replication Strategies
18 H4: syncrepl replication
21 H4: delta-syncrepl replication
27 H4: Working with Firewalls
36 H3: syncrepl replication
39 H3: delta-syncrepl replication
42 H3: N-Way Multi-Master
48 H2: LDAP Sync Replication
50 The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
51 short, is a consumer-side replication engine that enables the
52 consumer {{TERM:LDAP}} server to maintain a shadow copy of a
53 {{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
54 as one of the {{slapd}}(8) threads. It creates and maintains a
55 consumer replica by connecting to the replication provider to perform
56 the initial DIT content load followed either by periodic content
57 polling or by timely updates upon content changes.
59 Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
60 short) protocol as the replica synchronization protocol. It provides
61 a stateful replication which supports both pull-based and push-based
62 synchronization and does not mandate the use of a history store.
64 Syncrepl keeps track of the status of the replication content by
65 maintaining and exchanging synchronization cookies. Because the
66 syncrepl consumer and provider maintain their content status, the
67 consumer can poll the provider content to perform incremental
68 synchronization by asking for the entries required to make the
69 consumer replica up-to-date with the provider content. Syncrepl
70 also enables convenient management of replicas by maintaining replica
71 status. The consumer replica can be constructed from a consumer-side
72 or a provider-side backup at any synchronization status. Syncrepl
73 can automatically resynchronize the consumer replica up-to-date
74 with the current provider content.
76 Syncrepl supports both pull-based and push-based synchronization.
77 In its basic refreshOnly synchronization mode, the provider uses
78 pull-based synchronization where the consumer servers need not be
79 tracked and no history information is maintained. The information
80 required for the provider to process periodic polling requests is
81 contained in the synchronization cookie of the request itself. To
82 optimize the pull-based synchronization, syncrepl utilizes the
83 present phase of the LDAP Sync protocol as well as its delete phase,
84 instead of falling back on frequent full reloads. To further optimize
85 the pull-based synchronization, the provider can maintain a per-scope
86 session log as a history store. In its refreshAndPersist mode of
87 synchronization, the provider uses a push-based synchronization.
88 The provider keeps track of the consumer servers that have requested
89 a persistent search and sends them necessary updates as the provider
90 replication content gets modified.
92 With syncrepl, a consumer server can create a replica without
93 changing the provider's configurations and without restarting the
94 provider server, if the consumer server has appropriate access
95 privileges for the DIT fragment to be replicated. The consumer
96 server can stop the replication also without the need for provider-side
99 Syncrepl supports both partial and sparse replications. The shadow
100 DIT fragment is defined by a general search criteria consisting of
101 base, scope, filter, and attribute list. The replica content is
102 also subject to the access privileges of the bind identity of the
103 syncrepl replication connection.
106 H3: The LDAP Content Synchronization Protocol
108 The LDAP Sync protocol allows a client to maintain a synchronized
109 copy of a DIT fragment. The LDAP Sync operation is defined as a set
110 of controls and other protocol elements which extend the LDAP search
111 operation. This section introduces the LDAP Content Sync protocol
112 only briefly. For more information, refer to {{REF:RFC4533}}.
114 The LDAP Sync protocol supports both polling and listening for
115 changes by defining two respective synchronization operations:
116 {{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
117 by the {{refreshOnly}} operation. The client copy is synchronized
118 to the server copy at the time of polling. The server finishes the
119 search operation by returning {{SearchResultDone}} at the end of
120 the search operation as in the normal search. The listening is
121 implemented by the {{refreshAndPersist}} operation. Instead of
122 finishing the search after returning all entries currently matching
123 the search criteria, the synchronization search remains persistent
124 in the server. Subsequent updates to the synchronization content
125 in the server cause additional entry updates to be sent to the
128 The {{refreshOnly}} operation and the refresh stage of the
129 {{refreshAndPersist}} operation can be performed with a present
130 phase or a delete phase.
132 In the present phase, the server sends the client the entries updated
133 within the search scope since the last synchronization. The server
134 sends all requested attributes, be it changed or not, of the updated
135 entries. For each unchanged entry which remains in the scope, the
136 server sends a present message consisting only of the name of the
137 entry and the synchronization control representing state present.
138 The present message does not contain any attributes of the entry.
139 After the client receives all update and present entries, it can
140 reliably determine the new client copy by adding the entries added
141 to the server, by replacing the entries modified at the server, and
142 by deleting entries in the client copy which have not been updated
143 nor specified as being present at the server.
145 The transmission of the updated entries in the delete phase is the
146 same as in the present phase. The server sends all the requested
147 attributes of the entries updated within the search scope since the
148 last synchronization to the client. In the delete phase, however,
149 the server sends a delete message for each entry deleted from the
150 search scope, instead of sending present messages. The delete
151 message consists only of the name of the entry and the synchronization
152 control representing state delete. The new client copy can be
153 determined by adding, modifying, and removing entries according to
154 the synchronization control attached to the {{SearchResultEntry}}
157 In the case that the LDAP Sync server maintains a history store and
158 can determine which entries are scoped out of the client copy since
159 the last synchronization time, the server can use the delete phase.
160 If the server does not maintain any history store, cannot determine
161 the scoped-out entries from the history store, or the history store
162 does not cover the outdated synchronization state of the client,
163 the server should use the present phase. The use of the present
164 phase is much more efficient than a full content reload in terms
165 of the synchronization traffic. To reduce the synchronization
166 traffic further, the LDAP Sync protocol also provides several
167 optimizations such as the transmission of the normalized {{EX:entryUUID}}s
168 and the transmission of multiple {{EX:entryUUIDs}} in a single
169 {{syncIdSet}} message.
171 At the end of the {{refreshOnly}} synchronization, the server sends
172 a synchronization cookie to the client as a state indicator of the
173 client copy after the synchronization is completed. The client
174 will present the received cookie when it requests the next incremental
175 synchronization to the server.
177 When {{refreshAndPersist}} synchronization is used, the server sends
178 a synchronization cookie at the end of the refresh stage by sending
179 a Sync Info message with TRUE refreshDone. It also sends a
180 synchronization cookie by attaching it to {{SearchResultEntry}}
181 generated in the persist stage of the synchronization search. During
182 the persist stage, the server can also send a Sync Info message
183 containing the synchronization cookie at any time the server wants
184 to update the client-side state indicator. The server also updates
185 a synchronization indicator of the client at the end of the persist
188 In the LDAP Sync protocol, entries are uniquely identified by the
189 {{EX:entryUUID}} attribute value. It can function as a reliable
190 identifier of the entry. The DN of the entry, on the other hand,
191 can be changed over time and hence cannot be considered as the
192 reliable identifier. The {{EX:entryUUID}} is attached to each
193 {{SearchResultEntry}} or {{SearchResultReference}} as a part of the
194 synchronization control.
199 The syncrepl engine utilizes both the {{refreshOnly}} and the
200 {{refreshAndPersist}} operations of the LDAP Sync protocol. If a
201 syncrepl specification is included in a database definition,
202 {{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
203 and schedules its execution. If the {{refreshOnly}} operation is
204 specified, the syncrepl engine will be rescheduled at the interval
205 time after a synchronization operation is completed. If the
206 {{refreshAndPersist}} operation is specified, the engine will remain
207 active and process the persistent synchronization messages from the
210 The syncrepl engine utilizes both the present phase and the delete
211 phase of the refresh synchronization. It is possible to configure
212 a per-scope session log in the provider server which stores the
213 {{EX:entryUUID}}s of a finite number of entries deleted from a
214 replication content. Multiple replicas of single provider content
215 share the same per-scope session log. The syncrepl engine uses the
216 delete phase if the session log is present and the state of the
217 consumer server is recent enough that no session log entries are
218 truncated after the last synchronization of the client. The syncrepl
219 engine uses the present phase if no session log is configured for
220 the replication content or if the consumer replica is too outdated
221 to be covered by the session log. The current design of the session
222 log store is memory based, so the information contained in the
223 session log is not persistent over multiple provider invocations.
224 It is not currently supported to access the session log store by
225 using LDAP operations. It is also not currently supported to impose
226 access control to the session log.
228 As a further optimization, even in the case the synchronization
229 search is not associated with any session log, no entries will be
230 transmitted to the consumer server when there has been no update
231 in the replication context.
233 The syncrepl engine, which is a consumer-side replication engine,
234 can work with any backends. The LDAP Sync provider can be configured
235 as an overlay on any backend, but works best with the {{back-bdb}}
236 or {{back-hdb}} backend.
238 The LDAP Sync provider maintains a {{EX:contextCSN}} for each
239 database as the current synchronization state indicator of the
240 provider content. It is the largest {{EX:entryCSN}} in the provider
241 context such that no transactions for an entry having smaller
242 {{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}}
243 could not just be set to the largest issued {{EX:entryCSN}} because
244 {{EX:entryCSN}} is obtained before a transaction starts and
245 transactions are not committed in the issue order.
247 The provider stores the {{EX:contextCSN}} of a context in the
248 {{EX:contextCSN}} attribute of the context suffix entry. The attribute
249 is not written to the database after every update operation though;
250 instead it is maintained primarily in memory. At database start
251 time the provider reads the last saved {{EX:contextCSN}} into memory
252 and uses the in-memory copy exclusively thereafter. By default,
253 changes to the {{EX:contextCSN}} as a result of database updates
254 will not be written to the database until the server is cleanly
255 shut down. A checkpoint facility exists to cause the contextCSN to
256 be written out more frequently if desired.
258 Note that at startup time, if the provider is unable to read a
259 {{EX:contextCSN}} from the suffix entry, it will scan the entire
260 database to determine the value, and this scan may take quite a
261 long time on a large database. When a {{EX:contextCSN}} value is
262 read, the database will still be scanned for any {{EX:entryCSN}}
263 values greater than it, to make sure the {{EX:contextCSN}} value
264 truly reflects the greatest committed {{EX:entryCSN}} in the database.
265 On databases which support inequality indexing, setting an eq index
266 on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
267 checkpoints will greatly speed up this scanning step.
269 If no {{EX:contextCSN}} can be determined by reading and scanning
270 the database, a new value will be generated. Also, if scanning the
271 database yielded a greater {{EX:entryCSN}} than was previously
272 recorded in the suffix entry's {{EX:contextCSN}} attribute, a
273 checkpoint will be immediately written with the new value.
275 The consumer also stores its replica state, which is the provider's
276 {{EX:contextCSN}} received as a synchronization cookie, in the
277 {{EX:contextCSN}} attribute of the suffix entry. The replica state
278 maintained by a consumer server is used as the synchronization state
279 indicator when it performs subsequent incremental synchronization
280 with the provider server. It is also used as a provider-side
281 synchronization state indicator when it functions as a secondary
282 provider server in a cascading replication configuration. Since
283 the consumer and provider state information are maintained in the
284 same location within their respective databases, any consumer can
285 be promoted to a provider (and vice versa) without any special
288 Because a general search filter can be used in the syncrepl
289 specification, some entries in the context may be omitted from the
290 synchronization content. The syncrepl engine creates a glue entry
291 to fill in the holes in the replica context if any part of the
292 replica content is subordinate to the holes. The glue entries will
293 not be returned in the search result unless {{ManageDsaIT}} control
296 Also as a consequence of the search filter used in the syncrepl
297 specification, it is possible for a modification to remove an entry
298 from the replication scope even though the entry has not been deleted
299 on the provider. Logically the entry must be deleted on the consumer
300 but in {{refreshOnly}} mode the provider cannot detect and propagate
301 this change without the use of the session log.
304 H3: Configuring Syncrepl
306 Because syncrepl is a consumer-side replication engine, the syncrepl
307 specification is defined in {{slapd.conf}}(5) of the consumer
308 server, not in the provider server's configuration file. The initial
309 loading of the replica content can be performed either by starting
310 the syncrepl engine with no synchronization cookie or by populating
311 the consumer replica by adding an {{TERM:LDIF}} file dumped as a
312 backup at the provider.
314 When loading from a backup, it is not required to perform the initial
315 loading from the up-to-date backup of the provider content. The
316 syncrepl engine will automatically synchronize the initial consumer
317 replica to the current provider content. As a result, it is not
318 required to stop the provider server in order to avoid the replica
319 inconsistency caused by the updates to the provider content during
320 the content backup and loading process.
322 When replicating a large scale directory, especially in a bandwidth
323 constrained environment, it is advised to load the consumer replica
324 from a backup instead of performing a full initial load using
328 H4: Set up the provider slapd
330 The provider is implemented as an overlay, so the overlay itself
331 must first be configured in {{slapd.conf}}(5) before it can be
332 used. The provider has only two configuration directives, for setting
333 checkpoints on the {{EX:contextCSN}} and for configuring the session
334 log. Because the LDAP Sync search is subject to access control,
335 proper access control privileges should be set up for the replicated
338 The {{EX:contextCSN}} checkpoint is configured by the
340 > syncprov-checkpoint <ops> <minutes>
342 directive. Checkpoints are only tested after successful write
343 operations. If {{<ops>}} operations or more than {{<minutes>}}
344 time has passed since the last checkpoint, a new checkpoint is
347 The session log is configured by the
349 > syncprov-sessionlog <size>
351 directive, where {{<size>}} is the maximum number of session log
352 entries the session log can record. When a session log is configured,
353 it is automatically used for all LDAP Sync searches within the
356 Note that using the session log requires searching on the {{entryUUID}}
357 attribute. Setting an eq index on this attribute will greatly benefit
358 the performance of the session log on the provider.
360 A more complete example of the {{slapd.conf}}(5) content is thus:
363 > suffix dc=Example,dc=com
364 > rootdn dc=Example,dc=com
365 > directory /var/ldap/db
366 > index objectclass,entryCSN,entryUUID eq
369 > syncprov-checkpoint 100 10
370 > syncprov-sessionlog 100
373 H4: Set up the consumer slapd
375 The syncrepl replication is specified in the database section of
376 {{slapd.conf}}(5) for the replica context. The syncrepl engine
377 is backend independent and the directive can be defined with any
381 > suffix dc=Example,dc=com
382 > rootdn dc=Example,dc=com
383 > directory /var/ldap/db
384 > index objectclass,entryCSN,entryUUID eq
387 > provider=ldap://provider.example.com:389
389 > interval=01:00:00:00
390 > searchbase="dc=example,dc=com"
391 > filter="(objectClass=organizationalPerson)"
393 > attrs="cn,sn,ou,telephoneNumber,title,l"
396 > binddn="cn=syncuser,dc=example,dc=com"
399 In this example, the consumer will connect to the provider {{slapd}}(8)
400 at port 389 of {{FILE:ldap://provider.example.com}} to perform a
401 polling ({{refreshOnly}}) mode of synchronization once a day. It
402 will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
403 authentication with password "secret". Note that the access control
404 privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
405 appropriately in the provider to retrieve the desired replication
406 content. Also the search limits must be high enough on the provider
407 to allow the syncuser to retrieve a complete copy of the requested
408 content. The consumer uses the rootdn to write to its database so
409 it always has full permissions to write all content.
411 The synchronization search in the above example will search for the
412 entries whose objectClass is organizationalPerson in the entire
413 subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
414 are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
415 {{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
416 that the consumer {{slapd}}(8) will not enforce entry schema
417 checking when it process updates from the provider {{slapd}}(8).
419 For more detailed information on the syncrepl directive, see the
420 {{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
421 chapter of this admin guide.
424 H4: Start the provider and the consumer slapd
426 The provider {{slapd}}(8) is not required to be restarted.
427 {{contextCSN}} is automatically generated as needed: it might be
428 originally contained in the {{TERM:LDIF}} file, generated by
429 {{slapadd}} (8), generated upon changes in the context, or generated
430 when the first LDAP Sync search arrives at the provider. If an
431 LDIF file is being loaded which did not previously contain the
432 {{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
433 (8) to cause it to be generated. This will allow the server to
434 startup a little quicker the first time it runs.
436 When starting a consumer {{slapd}}(8), it is possible to provide
437 a synchronization cookie as the {{-c cookie}} command line option
438 in order to start the synchronization from a specific state. The
439 cookie is a comma separated list of name=value pairs. Currently
440 supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
441 {{<csn>}} represents the current synchronization state of the
442 consumer replica. {{<rid>}} identifies a consumer replica locally
443 within the consumer server. It is used to relate the cookie to the
444 syncrepl definition in {{slapd.conf}}(5) which has the matching
445 replica identifier. The {{<rid>}} must have no more than 3 decimal
446 digits. The command line cookie overrides the synchronization
447 cookie stored in the consumer replica database.
450 H2: N-Way Multi-Master
projects / openldap / blob