2 # Copyright 1999-2007 The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
8 H2: Replication Strategies
11 H3: Working with Firewalls
17 H3: syncrepl replication
20 H3: delta-syncrepl replication
23 H3: N-Way Multi-Master
29 H2: LDAP Sync Replication
31 The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
32 short, is a consumer-side replication engine that enables the
33 consumer {{TERM:LDAP}} server to maintain a shadow copy of a
34 {{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
35 as one of the {{slapd}}(8) threads. It creates and maintains a
36 consumer replica by connecting to the replication provider to perform
37 the initial DIT content load followed either by periodic content
38 polling or by timely updates upon content changes.
40 Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
41 short) protocol as the replica synchronization protocol. It provides
42 a stateful replication which supports both pull-based and push-based
43 synchronization and does not mandate the use of a history store.
45 Syncrepl keeps track of the status of the replication content by
46 maintaining and exchanging synchronization cookies. Because the
47 syncrepl consumer and provider maintain their content status, the
48 consumer can poll the provider content to perform incremental
49 synchronization by asking for the entries required to make the
50 consumer replica up-to-date with the provider content. Syncrepl
51 also enables convenient management of replicas by maintaining replica
52 status. The consumer replica can be constructed from a consumer-side
53 or a provider-side backup at any synchronization status. Syncrepl
54 can automatically resynchronize the consumer replica up-to-date
55 with the current provider content.
57 Syncrepl supports both pull-based and push-based synchronization.
58 In its basic refreshOnly synchronization mode, the provider uses
59 pull-based synchronization where the consumer servers need not be
60 tracked and no history information is maintained. The information
61 required for the provider to process periodic polling requests is
62 contained in the synchronization cookie of the request itself. To
63 optimize the pull-based synchronization, syncrepl utilizes the
64 present phase of the LDAP Sync protocol as well as its delete phase,
65 instead of falling back on frequent full reloads. To further optimize
66 the pull-based synchronization, the provider can maintain a per-scope
67 session log as a history store. In its refreshAndPersist mode of
68 synchronization, the provider uses a push-based synchronization.
69 The provider keeps track of the consumer servers that have requested
70 a persistent search and sends them necessary updates as the provider
71 replication content gets modified.
73 With syncrepl, a consumer server can create a replica without
74 changing the provider's configurations and without restarting the
75 provider server, if the consumer server has appropriate access
76 privileges for the DIT fragment to be replicated. The consumer
77 server can stop the replication also without the need for provider-side
80 Syncrepl supports both partial and sparse replications. The shadow
81 DIT fragment is defined by a general search criteria consisting of
82 base, scope, filter, and attribute list. The replica content is
83 also subject to the access privileges of the bind identity of the
84 syncrepl replication connection.
87 H3: The LDAP Content Synchronization Protocol
89 The LDAP Sync protocol allows a client to maintain a synchronized
90 copy of a DIT fragment. The LDAP Sync operation is defined as a set
91 of controls and other protocol elements which extend the LDAP search
92 operation. This section introduces the LDAP Content Sync protocol
93 only briefly. For more information, refer to {{REF:RFC4533}}.
95 The LDAP Sync protocol supports both polling and listening for
96 changes by defining two respective synchronization operations:
97 {{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
98 by the {{refreshOnly}} operation. The client copy is synchronized
99 to the server copy at the time of polling. The server finishes the
100 search operation by returning {{SearchResultDone}} at the end of
101 the search operation as in the normal search. The listening is
102 implemented by the {{refreshAndPersist}} operation. Instead of
103 finishing the search after returning all entries currently matching
104 the search criteria, the synchronization search remains persistent
105 in the server. Subsequent updates to the synchronization content
106 in the server cause additional entry updates to be sent to the
109 The {{refreshOnly}} operation and the refresh stage of the
110 {{refreshAndPersist}} operation can be performed with a present
111 phase or a delete phase.
113 In the present phase, the server sends the client the entries updated
114 within the search scope since the last synchronization. The server
115 sends all requested attributes, be it changed or not, of the updated
116 entries. For each unchanged entry which remains in the scope, the
117 server sends a present message consisting only of the name of the
118 entry and the synchronization control representing state present.
119 The present message does not contain any attributes of the entry.
120 After the client receives all update and present entries, it can
121 reliably determine the new client copy by adding the entries added
122 to the server, by replacing the entries modified at the server, and
123 by deleting entries in the client copy which have not been updated
124 nor specified as being present at the server.
126 The transmission of the updated entries in the delete phase is the
127 same as in the present phase. The server sends all the requested
128 attributes of the entries updated within the search scope since the
129 last synchronization to the client. In the delete phase, however,
130 the server sends a delete message for each entry deleted from the
131 search scope, instead of sending present messages. The delete
132 message consists only of the name of the entry and the synchronization
133 control representing state delete. The new client copy can be
134 determined by adding, modifying, and removing entries according to
135 the synchronization control attached to the {{SearchResultEntry}}
138 In the case that the LDAP Sync server maintains a history store and
139 can determine which entries are scoped out of the client copy since
140 the last synchronization time, the server can use the delete phase.
141 If the server does not maintain any history store, cannot determine
142 the scoped-out entries from the history store, or the history store
143 does not cover the outdated synchronization state of the client,
144 the server should use the present phase. The use of the present
145 phase is much more efficient than a full content reload in terms
146 of the synchronization traffic. To reduce the synchronization
147 traffic further, the LDAP Sync protocol also provides several
148 optimizations such as the transmission of the normalized {{EX:entryUUID}}s
149 and the transmission of multiple {{EX:entryUUIDs}} in a single
150 {{syncIdSet}} message.
152 At the end of the {{refreshOnly}} synchronization, the server sends
153 a synchronization cookie to the client as a state indicator of the
154 client copy after the synchronization is completed. The client
155 will present the received cookie when it requests the next incremental
156 synchronization to the server.
158 When {{refreshAndPersist}} synchronization is used, the server sends
159 a synchronization cookie at the end of the refresh stage by sending
160 a Sync Info message with TRUE refreshDone. It also sends a
161 synchronization cookie by attaching it to {{SearchResultEntry}}
162 generated in the persist stage of the synchronization search. During
163 the persist stage, the server can also send a Sync Info message
164 containing the synchronization cookie at any time the server wants
165 to update the client-side state indicator. The server also updates
166 a synchronization indicator of the client at the end of the persist
169 In the LDAP Sync protocol, entries are uniquely identified by the
170 {{EX:entryUUID}} attribute value. It can function as a reliable
171 identifier of the entry. The DN of the entry, on the other hand,
172 can be changed over time and hence cannot be considered as the
173 reliable identifier. The {{EX:entryUUID}} is attached to each
174 {{SearchResultEntry}} or {{SearchResultReference}} as a part of the
175 synchronization control.
180 The syncrepl engine utilizes both the {{refreshOnly}} and the
181 {{refreshAndPersist}} operations of the LDAP Sync protocol. If a
182 syncrepl specification is included in a database definition,
183 {{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
184 and schedules its execution. If the {{refreshOnly}} operation is
185 specified, the syncrepl engine will be rescheduled at the interval
186 time after a synchronization operation is completed. If the
187 {{refreshAndPersist}} operation is specified, the engine will remain
188 active and process the persistent synchronization messages from the
191 The syncrepl engine utilizes both the present phase and the delete
192 phase of the refresh synchronization. It is possible to configure
193 a per-scope session log in the provider server which stores the
194 {{EX:entryUUID}}s of a finite number of entries deleted from a
195 replication content. Multiple replicas of single provider content
196 share the same per-scope session log. The syncrepl engine uses the
197 delete phase if the session log is present and the state of the
198 consumer server is recent enough that no session log entries are
199 truncated after the last synchronization of the client. The syncrepl
200 engine uses the present phase if no session log is configured for
201 the replication content or if the consumer replica is too outdated
202 to be covered by the session log. The current design of the session
203 log store is memory based, so the information contained in the
204 session log is not persistent over multiple provider invocations.
205 It is not currently supported to access the session log store by
206 using LDAP operations. It is also not currently supported to impose
207 access control to the session log.
209 As a further optimization, even in the case the synchronization
210 search is not associated with any session log, no entries will be
211 transmitted to the consumer server when there has been no update
212 in the replication context.
214 The syncrepl engine, which is a consumer-side replication engine,
215 can work with any backends. The LDAP Sync provider can be configured
216 as an overlay on any backend, but works best with the {{back-bdb}}
217 or {{back-hdb}} backend.
219 The LDAP Sync provider maintains a {{EX:contextCSN}} for each
220 database as the current synchronization state indicator of the
221 provider content. It is the largest {{EX:entryCSN}} in the provider
222 context such that no transactions for an entry having smaller
223 {{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}}
224 could not just be set to the largest issued {{EX:entryCSN}} because
225 {{EX:entryCSN}} is obtained before a transaction starts and
226 transactions are not committed in the issue order.
228 The provider stores the {{EX:contextCSN}} of a context in the
229 {{EX:contextCSN}} attribute of the context suffix entry. The attribute
230 is not written to the database after every update operation though;
231 instead it is maintained primarily in memory. At database start
232 time the provider reads the last saved {{EX:contextCSN}} into memory
233 and uses the in-memory copy exclusively thereafter. By default,
234 changes to the {{EX:contextCSN}} as a result of database updates
235 will not be written to the database until the server is cleanly
236 shut down. A checkpoint facility exists to cause the contextCSN to
237 be written out more frequently if desired.
239 Note that at startup time, if the provider is unable to read a
240 {{EX:contextCSN}} from the suffix entry, it will scan the entire
241 database to determine the value, and this scan may take quite a
242 long time on a large database. When a {{EX:contextCSN}} value is
243 read, the database will still be scanned for any {{EX:entryCSN}}
244 values greater than it, to make sure the {{EX:contextCSN}} value
245 truly reflects the greatest committed {{EX:entryCSN}} in the database.
246 On databases which support inequality indexing, setting an eq index
247 on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
248 checkpoints will greatly speed up this scanning step.
250 If no {{EX:contextCSN}} can be determined by reading and scanning
251 the database, a new value will be generated. Also, if scanning the
252 database yielded a greater {{EX:entryCSN}} than was previously
253 recorded in the suffix entry's {{EX:contextCSN}} attribute, a
254 checkpoint will be immediately written with the new value.
256 The consumer also stores its replica state, which is the provider's
257 {{EX:contextCSN}} received as a synchronization cookie, in the
258 {{EX:contextCSN}} attribute of the suffix entry. The replica state
259 maintained by a consumer server is used as the synchronization state
260 indicator when it performs subsequent incremental synchronization
261 with the provider server. It is also used as a provider-side
262 synchronization state indicator when it functions as a secondary
263 provider server in a cascading replication configuration. Since
264 the consumer and provider state information are maintained in the
265 same location within their respective databases, any consumer can
266 be promoted to a provider (and vice versa) without any special
269 Because a general search filter can be used in the syncrepl
270 specification, some entries in the context may be omitted from the
271 synchronization content. The syncrepl engine creates a glue entry
272 to fill in the holes in the replica context if any part of the
273 replica content is subordinate to the holes. The glue entries will
274 not be returned in the search result unless {{ManageDsaIT}} control
277 Also as a consequence of the search filter used in the syncrepl
278 specification, it is possible for a modification to remove an entry
279 from the replication scope even though the entry has not been deleted
280 on the provider. Logically the entry must be deleted on the consumer
281 but in {{refreshOnly}} mode the provider cannot detect and propagate
282 this change without the use of the session log.
285 H3: Configuring Syncrepl
287 Because syncrepl is a consumer-side replication engine, the syncrepl
288 specification is defined in {{slapd.conf}}(5) of the consumer
289 server, not in the provider server's configuration file. The initial
290 loading of the replica content can be performed either by starting
291 the syncrepl engine with no synchronization cookie or by populating
292 the consumer replica by adding an {{TERM:LDIF}} file dumped as a
293 backup at the provider.
295 When loading from a backup, it is not required to perform the initial
296 loading from the up-to-date backup of the provider content. The
297 syncrepl engine will automatically synchronize the initial consumer
298 replica to the current provider content. As a result, it is not
299 required to stop the provider server in order to avoid the replica
300 inconsistency caused by the updates to the provider content during
301 the content backup and loading process.
303 When replicating a large scale directory, especially in a bandwidth
304 constrained environment, it is advised to load the consumer replica
305 from a backup instead of performing a full initial load using
309 H4: Set up the provider slapd
311 The provider is implemented as an overlay, so the overlay itself
312 must first be configured in {{slapd.conf}}(5) before it can be
313 used. The provider has only two configuration directives, for setting
314 checkpoints on the {{EX:contextCSN}} and for configuring the session
315 log. Because the LDAP Sync search is subject to access control,
316 proper access control privileges should be set up for the replicated
319 The {{EX:contextCSN}} checkpoint is configured by the
321 > syncprov-checkpoint <ops> <minutes>
323 directive. Checkpoints are only tested after successful write
324 operations. If {{<ops>}} operations or more than {{<minutes>}}
325 time has passed since the last checkpoint, a new checkpoint is
328 The session log is configured by the
330 > syncprov-sessionlog <size>
332 directive, where {{<size>}} is the maximum number of session log
333 entries the session log can record. When a session log is configured,
334 it is automatically used for all LDAP Sync searches within the
337 Note that using the session log requires searching on the {{entryUUID}}
338 attribute. Setting an eq index on this attribute will greatly benefit
339 the performance of the session log on the provider.
341 A more complete example of the {{slapd.conf}}(5) content is thus:
344 > suffix dc=Example,dc=com
345 > rootdn dc=Example,dc=com
346 > directory /var/ldap/db
347 > index objectclass,entryCSN,entryUUID eq
350 > syncprov-checkpoint 100 10
351 > syncprov-sessionlog 100
354 H4: Set up the consumer slapd
356 The syncrepl replication is specified in the database section of
357 {{slapd.conf}}(5) for the replica context. The syncrepl engine
358 is backend independent and the directive can be defined with any
362 > suffix dc=Example,dc=com
363 > rootdn dc=Example,dc=com
364 > directory /var/ldap/db
365 > index objectclass,entryCSN,entryUUID eq
368 > provider=ldap://provider.example.com:389
370 > interval=01:00:00:00
371 > searchbase="dc=example,dc=com"
372 > filter="(objectClass=organizationalPerson)"
374 > attrs="cn,sn,ou,telephoneNumber,title,l"
377 > binddn="cn=syncuser,dc=example,dc=com"
380 In this example, the consumer will connect to the provider {{slapd}}(8)
381 at port 389 of {{FILE:ldap://provider.example.com}} to perform a
382 polling ({{refreshOnly}}) mode of synchronization once a day. It
383 will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
384 authentication with password "secret". Note that the access control
385 privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
386 appropriately in the provider to retrieve the desired replication
387 content. Also the search limits must be high enough on the provider
388 to allow the syncuser to retrieve a complete copy of the requested
389 content. The consumer uses the rootdn to write to its database so
390 it always has full permissions to write all content.
392 The synchronization search in the above example will search for the
393 entries whose objectClass is organizationalPerson in the entire
394 subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
395 are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
396 {{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
397 that the consumer {{slapd}}(8) will not enforce entry schema
398 checking when it process updates from the provider {{slapd}}(8).
400 For more detailed information on the syncrepl directive, see the
401 {{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
402 chapter of this admin guide.
405 H4: Start the provider and the consumer slapd
407 The provider {{slapd}}(8) is not required to be restarted.
408 {{contextCSN}} is automatically generated as needed: it might be
409 originally contained in the {{TERM:LDIF}} file, generated by
410 {{slapadd}} (8), generated upon changes in the context, or generated
411 when the first LDAP Sync search arrives at the provider. If an
412 LDIF file is being loaded which did not previously contain the
413 {{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
414 (8) to cause it to be generated. This will allow the server to
415 startup a little quicker the first time it runs.
417 When starting a consumer {{slapd}}(8), it is possible to provide
418 a synchronization cookie as the {{-c cookie}} command line option
419 in order to start the synchronization from a specific state. The
420 cookie is a comma separated list of name=value pairs. Currently
421 supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
422 {{<csn>}} represents the current synchronization state of the
423 consumer replica. {{<rid>}} identifies a consumer replica locally
424 within the consumer server. It is used to relate the cookie to the
425 syncrepl definition in {{slapd.conf}}(5) which has the matching
426 replica identifier. The {{<rid>}} must have no more than 3 decimal
427 digits. The command line cookie overrides the synchronization
428 cookie stored in the consumer replica database.
431 H2: N-Way Multi-Master
projects / openldap / blob