2 # Copyright 2003-2007 The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
5 H1: LDAP Sync Replication
7 The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
8 short, is a consumer-side replication engine that enables the
9 consumer {{TERM:LDAP}} server to maintain a shadow copy of a
10 {{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
11 as one of the {{slapd}}(8) threads. It creates and maintains a
12 consumer replica by connecting to the replication provider to perform
13 the initial DIT content load followed either by periodic content
14 polling or by timely updates upon content changes.
16 Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
17 short) protocol as the replica synchronization protocol. It provides
18 a stateful replication which supports both pull-based and push-based
19 synchronization and does not mandate the use of a history store.
21 Syncrepl keeps track of the status of the replication content by
22 maintaining and exchanging synchronization cookies. Because the
23 syncrepl consumer and provider maintain their content status, the
24 consumer can poll the provider content to perform incremental
25 synchronization by asking for the entries required to make the
26 consumer replica up-to-date with the provider content. Syncrepl
27 also enables convenient management of replicas by maintaining replica
28 status. The consumer replica can be constructed from a consumer-side
29 or a provider-side backup at any synchronization status. Syncrepl
30 can automatically resynchronize the consumer replica up-to-date
31 with the current provider content.
33 Syncrepl supports both pull-based and push-based synchronization.
34 In its basic refreshOnly synchronization mode, the provider uses
35 pull-based synchronization where the consumer servers need not be
36 tracked and no history information is maintained. The information
37 required for the provider to process periodic polling requests is
38 contained in the synchronization cookie of the request itself. To
39 optimize the pull-based synchronization, syncrepl utilizes the
40 present phase of the LDAP Sync protocol as well as its delete phase,
41 instead of falling back on frequent full reloads. To further optimize
42 the pull-based synchronization, the provider can maintain a per-scope
43 session log as a history store. In its refreshAndPersist mode of
44 synchronization, the provider uses a push-based synchronization.
45 The provider keeps track of the consumer servers that have requested
46 a persistent search and sends them necessary updates as the provider
47 replication content gets modified.
49 With syncrepl, a consumer server can create a replica without
50 changing the provider's configurations and without restarting the
51 provider server, if the consumer server has appropriate access
52 privileges for the DIT fragment to be replicated. The consumer
53 server can stop the replication also without the need for provider-side
56 Syncrepl supports both partial and sparse replications. The shadow
57 DIT fragment is defined by a general search criteria consisting of
58 base, scope, filter, and attribute list. The replica content is
59 also subject to the access privileges of the bind identity of the
60 syncrepl replication connection.
63 H2: The LDAP Content Synchronization Protocol
65 The LDAP Sync protocol allows a client to maintain a synchronized
66 copy of a DIT fragment. The LDAP Sync operation is defined as a set
67 of controls and other protocol elements which extend the LDAP search
68 operation. This section introduces the LDAP Content Sync protocol
69 only briefly. For more information, refer to {{REF:RFC4533}}.
71 The LDAP Sync protocol supports both polling and listening for
72 changes by defining two respective synchronization operations:
73 {{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
74 by the {{refreshOnly}} operation. The client copy is synchronized
75 to the server copy at the time of polling. The server finishes the
76 search operation by returning {{SearchResultDone}} at the end of
77 the search operation as in the normal search. The listening is
78 implemented by the {{refreshAndPersist}} operation. Instead of
79 finishing the search after returning all entries currently matching
80 the search criteria, the synchronization search remains persistent
81 in the server. Subsequent updates to the synchronization content
82 in the server cause additional entry updates to be sent to the
85 The {{refreshOnly}} operation and the refresh stage of the
86 {{refreshAndPersist}} operation can be performed with a present
87 phase or a delete phase.
89 In the present phase, the server sends the client the entries updated
90 within the search scope since the last synchronization. The server
91 sends all requested attributes, be it changed or not, of the updated
92 entries. For each unchanged entry which remains in the scope, the
93 server sends a present message consisting only of the name of the
94 entry and the synchronization control representing state present.
95 The present message does not contain any attributes of the entry.
96 After the client receives all update and present entries, it can
97 reliably determine the new client copy by adding the entries added
98 to the server, by replacing the entries modified at the server, and
99 by deleting entries in the client copy which have not been updated
100 nor specified as being present at the server.
102 The transmission of the updated entries in the delete phase is the
103 same as in the present phase. The server sends all the requested
104 attributes of the entries updated within the search scope since the
105 last synchronization to the client. In the delete phase, however,
106 the server sends a delete message for each entry deleted from the
107 search scope, instead of sending present messages. The delete
108 message consists only of the name of the entry and the synchronization
109 control representing state delete. The new client copy can be
110 determined by adding, modifying, and removing entries according to
111 the synchronization control attached to the {{SearchResultEntry}}
114 In the case that the LDAP Sync server maintains a history store and
115 can determine which entries are scoped out of the client copy since
116 the last synchronization time, the server can use the delete phase.
117 If the server does not maintain any history store, cannot determine
118 the scoped-out entries from the history store, or the history store
119 does not cover the outdated synchronization state of the client,
120 the server should use the present phase. The use of the present
121 phase is much more efficient than a full content reload in terms
122 of the synchronization traffic. To reduce the synchronization
123 traffic further, the LDAP Sync protocol also provides several
124 optimizations such as the transmission of the normalized {{EX:entryUUID}}s
125 and the transmission of multiple {{EX:entryUUIDs}} in a single
126 {{syncIdSet}} message.
128 At the end of the {{refreshOnly}} synchronization, the server sends
129 a synchronization cookie to the client as a state indicator of the
130 client copy after the synchronization is completed. The client
131 will present the received cookie when it requests the next incremental
132 synchronization to the server.
134 When {{refreshAndPersist}} synchronization is used, the server sends
135 a synchronization cookie at the end of the refresh stage by sending
136 a Sync Info message with TRUE refreshDone. It also sends a
137 synchronization cookie by attaching it to {{SearchResultEntry}}
138 generated in the persist stage of the synchronization search. During
139 the persist stage, the server can also send a Sync Info message
140 containing the synchronization cookie at any time the server wants
141 to update the client-side state indicator. The server also updates
142 a synchronization indicator of the client at the end of the persist
145 In the LDAP Sync protocol, entries are uniquely identified by the
146 {{EX:entryUUID}} attribute value. It can function as a reliable
147 identifier of the entry. The DN of the entry, on the other hand,
148 can be changed over time and hence cannot be considered as the
149 reliable identifier. The {{EX:entryUUID}} is attached to each
150 {{SearchResultEntry}} or {{SearchResultReference}} as a part of the
151 synchronization control.
156 The syncrepl engine utilizes both the {{refreshOnly}} and the
157 {{refreshAndPersist}} operations of the LDAP Sync protocol. If a
158 syncrepl specification is included in a database definition,
159 {{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
160 and schedules its execution. If the {{refreshOnly}} operation is
161 specified, the syncrepl engine will be rescheduled at the interval
162 time after a synchronization operation is completed. If the
163 {{refreshAndPersist}} operation is specified, the engine will remain
164 active and process the persistent synchronization messages from the
167 The syncrepl engine utilizes both the present phase and the delete
168 phase of the refresh synchronization. It is possible to configure
169 a per-scope session log in the provider server which stores the
170 {{EX:entryUUID}}s of a finite number of entries deleted from a
171 replication content. Multiple replicas of single provider content
172 share the same per-scope session log. The syncrepl engine uses the
173 delete phase if the session log is present and the state of the
174 consumer server is recent enough that no session log entries are
175 truncated after the last synchronization of the client. The syncrepl
176 engine uses the present phase if no session log is configured for
177 the replication content or if the consumer replica is too outdated
178 to be covered by the session log. The current design of the session
179 log store is memory based, so the information contained in the
180 session log is not persistent over multiple provider invocations.
181 It is not currently supported to access the session log store by
182 using LDAP operations. It is also not currently supported to impose
183 access control to the session log.
185 As a further optimization, even in the case the synchronization
186 search is not associated with any session log, no entries will be
187 transmitted to the consumer server when there has been no update
188 in the replication context.
190 The syncrepl engine, which is a consumer-side replication engine,
191 can work with any backends. The LDAP Sync provider can be configured
192 as an overlay on any backend, but works best with the {{back-bdb}}
193 or {{back-hdb}} backend.
195 The LDAP Sync provider maintains a {{EX:contextCSN}} for each
196 database as the current synchronization state indicator of the
197 provider content. It is the largest {{EX:entryCSN}} in the provider
198 context such that no transactions for an entry having smaller
199 {{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}}
200 could not just be set to the largest issued {{EX:entryCSN}} because
201 {{EX:entryCSN}} is obtained before a transaction starts and
202 transactions are not committed in the issue order.
204 The provider stores the {{EX:contextCSN}} of a context in the
205 {{EX:contextCSN}} attribute of the context suffix entry. The attribute
206 is not written to the database after every update operation though;
207 instead it is maintained primarily in memory. At database start
208 time the provider reads the last saved {{EX:contextCSN}} into memory
209 and uses the in-memory copy exclusively thereafter. By default,
210 changes to the {{EX:contextCSN}} as a result of database updates
211 will not be written to the database until the server is cleanly
212 shut down. A checkpoint facility exists to cause the contextCSN to
213 be written out more frequently if desired.
215 Note that at startup time, if the provider is unable to read a
216 {{EX:contextCSN}} from the suffix entry, it will scan the entire
217 database to determine the value, and this scan may take quite a
218 long time on a large database. When a {{EX:contextCSN}} value is
219 read, the database will still be scanned for any {{EX:entryCSN}}
220 values greater than it, to make sure the {{EX:contextCSN}} value
221 truly reflects the greatest committed {{EX:entryCSN}} in the database.
222 On databases which support inequality indexing, setting an eq index
223 on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
224 checkpoints will greatly speed up this scanning step.
226 If no {{EX:contextCSN}} can be determined by reading and scanning
227 the database, a new value will be generated. Also, if scanning the
228 database yielded a greater {{EX:entryCSN}} than was previously
229 recorded in the suffix entry's {{EX:contextCSN}} attribute, a
230 checkpoint will be immediately written with the new value.
232 The consumer also stores its replica state, which is the provider's
233 {{EX:contextCSN}} received as a synchronization cookie, in the
234 {{EX:contextCSN}} attribute of the suffix entry. The replica state
235 maintained by a consumer server is used as the synchronization state
236 indicator when it performs subsequent incremental synchronization
237 with the provider server. It is also used as a provider-side
238 synchronization state indicator when it functions as a secondary
239 provider server in a cascading replication configuration. Since
240 the consumer and provider state information are maintained in the
241 same location within their respective databases, any consumer can
242 be promoted to a provider (and vice versa) without any special
245 Because a general search filter can be used in the syncrepl
246 specification, some entries in the context may be omitted from the
247 synchronization content. The syncrepl engine creates a glue entry
248 to fill in the holes in the replica context if any part of the
249 replica content is subordinate to the holes. The glue entries will
250 not be returned in the search result unless {{ManageDsaIT}} control
253 Also as a consequence of the search filter used in the syncrepl
254 specification, it is possible for a modification to remove an entry
255 from the replication scope even though the entry has not been deleted
256 on the provider. Logically the entry must be deleted on the consumer
257 but in {{refreshOnly}} mode the provider cannot detect and propagate
258 this change without the use of the session log.
261 H2: Configuring Syncrepl
263 Because syncrepl is a consumer-side replication engine, the syncrepl
264 specification is defined in {{slapd.conf}}(5) of the consumer
265 server, not in the provider server's configuration file. The initial
266 loading of the replica content can be performed either by starting
267 the syncrepl engine with no synchronization cookie or by populating
268 the consumer replica by adding an {{TERM:LDIF}} file dumped as a
269 backup at the provider.
271 When loading from a backup, it is not required to perform the initial
272 loading from the up-to-date backup of the provider content. The
273 syncrepl engine will automatically synchronize the initial consumer
274 replica to the current provider content. As a result, it is not
275 required to stop the provider server in order to avoid the replica
276 inconsistency caused by the updates to the provider content during
277 the content backup and loading process.
279 When replicating a large scale directory, especially in a bandwidth
280 constrained environment, it is advised to load the consumer replica
281 from a backup instead of performing a full initial load using
285 H3: Set up the provider slapd
287 The provider is implemented as an overlay, so the overlay itself
288 must first be configured in {{slapd.conf}}(5) before it can be
289 used. The provider has only two configuration directives, for setting
290 checkpoints on the {{EX:contextCSN}} and for configuring the session
291 log. Because the LDAP Sync search is subject to access control,
292 proper access control privileges should be set up for the replicated
295 The {{EX:contextCSN}} checkpoint is configured by the
297 > syncprov-checkpoint <ops> <minutes>
299 directive. Checkpoints are only tested after successful write
300 operations. If {{<ops>}} operations or more than {{<minutes>}}
301 time has passed since the last checkpoint, a new checkpoint is
304 The session log is configured by the
306 > syncprov-sessionlog <size>
308 directive, where {{<size>}} is the maximum number of session log
309 entries the session log can record. When a session log is configured,
310 it is automatically used for all LDAP Sync searches within the
313 Note that using the session log requires searching on the {{entryUUID}}
314 attribute. Setting an eq index on this attribute will greatly benefit
315 the performance of the session log on the provider.
317 A more complete example of the {{slapd.conf}}(5) content is thus:
320 > suffix dc=Example,dc=com
321 > rootdn dc=Example,dc=com
322 > directory /var/ldap/db
323 > index objectclass,entryCSN,entryUUID eq
326 > syncprov-checkpoint 100 10
327 > syncprov-sessionlog 100
330 H3: Set up the consumer slapd
332 The syncrepl replication is specified in the database section of
333 {{slapd.conf}}(5) for the replica context. The syncrepl engine
334 is backend independent and the directive can be defined with any
338 > suffix dc=Example,dc=com
339 > rootdn dc=Example,dc=com
340 > directory /var/ldap/db
341 > index objectclass,entryCSN,entryUUID eq
344 > provider=ldap://provider.example.com:389
346 > interval=01:00:00:00
347 > searchbase="dc=example,dc=com"
348 > filter="(objectClass=organizationalPerson)"
350 > attrs="cn,sn,ou,telephoneNumber,title,l"
353 > binddn="cn=syncuser,dc=example,dc=com"
356 In this example, the consumer will connect to the provider {{slapd}}(8)
357 at port 389 of {{FILE:ldap://provider.example.com}} to perform a
358 polling ({{refreshOnly}}) mode of synchronization once a day. It
359 will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
360 authentication with password "secret". Note that the access control
361 privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
362 appropriately in the provider to retrieve the desired replication
363 content. Also the search limits must be high enough on the provider
364 to allow the syncuser to retrieve a complete copy of the requested
365 content. The consumer uses the rootdn to write to its database so
366 it always has full permissions to write all content.
368 The synchronization search in the above example will search for the
369 entries whose objectClass is organizationalPerson in the entire
370 subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
371 are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
372 {{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
373 that the consumer {{slapd}}(8) will not enforce entry schema
374 checking when it process updates from the provider {{slapd}}(8).
376 For more detailed information on the syncrepl directive, see the
377 {{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
378 chapter of this admin guide.
381 H3: Start the provider and the consumer slapd
383 The provider {{slapd}}(8) is not required to be restarted.
384 {{contextCSN}} is automatically generated as needed: it might be
385 originally contained in the {{TERM:LDIF}} file, generated by
386 {{slapadd}} (8), generated upon changes in the context, or generated
387 when the first LDAP Sync search arrives at the provider. If an
388 LDIF file is being loaded which did not previously contain the
389 {{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
390 (8) to cause it to be generated. This will allow the server to
391 startup a little quicker the first time it runs.
393 When starting a consumer {{slapd}}(8), it is possible to provide
394 a synchronization cookie as the {{-c cookie}} command line option
395 in order to start the synchronization from a specific state. The
396 cookie is a comma separated list of name=value pairs. Currently
397 supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
398 {{<csn>}} represents the current synchronization state of the
399 consumer replica. {{<rid>}} identifies a consumer replica locally
400 within the consumer server. It is used to relate the cookie to the
401 syncrepl definition in {{slapd.conf}}(5) which has the matching
402 replica identifier. The {{<rid>}} must have no more than 3 decimal
403 digits. The command line cookie overrides the synchronization
404 cookie stored in the consumer replica database.
projects / openldap / blob