2 # Copyright 1999-2009 The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
7 Replicated directories are a fundamental requirement for delivering a
8 resilient enterprise deployment.
10 {{PRD:OpenLDAP}} has various configuration options for creating a replicated
11 directory. In previous releases, replication was discussed in terms of
12 a {{master}} server and some number of {{slave}} servers. A master
13 accepted directory updates from other clients, and a slave only
14 accepted updates from a (single) master. The replication structure
15 was rigidly defined and any particular database could only fulfill
16 a single role, either master or slave.
18 As OpenLDAP now supports a wide variety of replication topologies, these
19 terms have been deprecated in favor of {{provider}} and
20 {{consumer}}: A provider replicates directory updates to consumers;
21 consumers receive replication updates from providers. Unlike the
22 rigidly defined master/slave relationships, provider/consumer roles
23 are quite fluid: replication updates received in a consumer can be
24 further propagated by that consumer to other servers, so a consumer
25 can also act simultaneously as a provider. Also, a consumer need not
26 be an actual LDAP server; it may be just an LDAP client.
28 The following sections will describe the replication technology and
29 discuss the various replication options that are available.
31 H2: Replication Technology
33 H3: LDAP Sync Replication
35 The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
36 short, is a consumer-side replication engine that enables the
37 consumer {{TERM:LDAP}} server to maintain a shadow copy of a
38 {{TERM:DIT}} fragment. A syncrepl engine resides at the consumer
39 and executes as one of the {{slapd}}(8) threads. It creates and maintains a
40 consumer replica by connecting to the replication provider to perform
41 the initial DIT content load followed either by periodic content
42 polling or by timely updates upon content changes.
44 Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for
45 short) as the replica synchronization protocol. LDAP Sync provides
46 a stateful replication which supports both pull-based and push-based
47 synchronization and does not mandate the use of a history store.
48 In pull-based replication the consumer periodically
49 polls the provider for updates. In push-based replication the consumer
50 listens for updates that are sent by the provider in realtime. Since the
51 protocol does not require a history store, the provider does not need to
52 maintain any log of updates it has received (Note
53 that the syncrepl engine is extensible and additional replication
54 protocols may be supported in the future.).
56 Syncrepl keeps track of the status of the replication content by
57 maintaining and exchanging synchronization cookies. Because the
58 syncrepl consumer and provider maintain their content status, the
59 consumer can poll the provider content to perform incremental
60 synchronization by asking for the entries required to make the
61 consumer replica up-to-date with the provider content. Syncrepl
62 also enables convenient management of replicas by maintaining replica
63 status. The consumer replica can be constructed from a consumer-side
64 or a provider-side backup at any synchronization status. Syncrepl
65 can automatically resynchronize the consumer replica up-to-date
66 with the current provider content.
68 Syncrepl supports both pull-based and push-based synchronization.
69 In its basic refreshOnly synchronization mode, the provider uses
70 pull-based synchronization where the consumer servers need not be
71 tracked and no history information is maintained. The information
72 required for the provider to process periodic polling requests is
73 contained in the synchronization cookie of the request itself. To
74 optimize the pull-based synchronization, syncrepl utilizes the
75 present phase of the LDAP Sync protocol as well as its delete phase,
76 instead of falling back on frequent full reloads. To further optimize
77 the pull-based synchronization, the provider can maintain a per-scope
78 session log as a history store. In its refreshAndPersist mode of
79 synchronization, the provider uses a push-based synchronization.
80 The provider keeps track of the consumer servers that have requested
81 a persistent search and sends them necessary updates as the provider
82 replication content gets modified.
84 With syncrepl, a consumer server can create a replica without
85 changing the provider's configurations and without restarting the
86 provider server, if the consumer server has appropriate access
87 privileges for the DIT fragment to be replicated. The consumer
88 server can stop the replication also without the need for provider-side
91 Syncrepl supports partial, sparse, and fractional replications. The shadow
92 DIT fragment is defined by a general search criteria consisting of
93 base, scope, filter, and attribute list. The replica content is
94 also subject to the access privileges of the bind identity of the
95 syncrepl replication connection.
98 H4: The LDAP Content Synchronization Protocol
100 The LDAP Sync protocol allows a client to maintain a synchronized
101 copy of a DIT fragment. The LDAP Sync operation is defined as a set
102 of controls and other protocol elements which extend the LDAP search
103 operation. This section introduces the LDAP Content Sync protocol
104 only briefly. For more information, refer to {{REF:RFC4533}}.
106 The LDAP Sync protocol supports both polling and listening for changes
107 by defining two respective synchronization operations:
108 {{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
109 by the {{refreshOnly}} operation. The consumer
110 polls the provider using an LDAP Search request with an LDAP Sync
111 control attached. The consumer copy is synchronized
112 to the provider copy at the time of polling using the information
113 returned in the search. The provider finishes the
114 search operation by returning {{SearchResultDone}} at the end of
115 the search operation as in the normal search. Listening is
116 implemented by the {{refreshAndPersist}} operation. As the name
117 implies, it begins with a search, like refreshOnly. Instead of
118 finishing the search after returning all entries currently matching
119 the search criteria, the synchronization search remains persistent
120 in the provider. Subsequent updates to the synchronization content
121 in the provider cause additional entry updates to be sent to the
124 The {{refreshOnly}} operation and the refresh stage of the
125 {{refreshAndPersist}} operation can be performed with a present
126 phase or a delete phase.
128 In the present phase, the provider sends the consumer the entries updated
129 within the search scope since the last synchronization. The provider
130 sends all requested attributes, be they changed or not, of the updated
131 entries. For each unchanged entry which remains in the scope, the
132 provider sends a present message consisting only of the name of the
133 entry and the synchronization control representing state present.
134 The present message does not contain any attributes of the entry.
135 After the consumer receives all update and present entries, it can
136 reliably determine the new consumer copy by adding the entries added
137 to the provider, by replacing the entries modified at the provider, and
138 by deleting entries in the consumer copy which have not been updated
139 nor specified as being present at the provider.
141 The transmission of the updated entries in the delete phase is the
142 same as in the present phase. The provider sends all the requested
143 attributes of the entries updated within the search scope since the
144 last synchronization to the consumer. In the delete phase, however,
145 the provider sends a delete message for each entry deleted from the
146 search scope, instead of sending present messages. The delete
147 message consists only of the name of the entry and the synchronization
148 control representing state delete. The new consumer copy can be
149 determined by adding, modifying, and removing entries according to
150 the synchronization control attached to the {{SearchResultEntry}}
153 In the case that the LDAP Sync provider maintains a history store and
154 can determine which entries are scoped out of the consumer copy since
155 the last synchronization time, the provider can use the delete phase.
156 If the provider does not maintain any history store, cannot determine
157 the scoped-out entries from the history store, or the history store
158 does not cover the outdated synchronization state of the consumer,
159 the provider should use the present phase. The use of the present
160 phase is much more efficient than a full content reload in terms
161 of the synchronization traffic. To reduce the synchronization
162 traffic further, the LDAP Sync protocol also provides several
163 optimizations such as the transmission of the normalized {{EX:entryUUID}}s
164 and the transmission of multiple {{EX:entryUUIDs}} in a single
165 {{syncIdSet}} message.
167 At the end of the {{refreshOnly}} synchronization, the provider sends
168 a synchronization cookie to the consumer as a state indicator of the
169 consumer copy after the synchronization is completed. The consumer
170 will present the received cookie when it requests the next incremental
171 synchronization to the provider.
173 When {{refreshAndPersist}} synchronization is used, the provider sends
174 a synchronization cookie at the end of the refresh stage by sending
175 a Sync Info message with refreshDone=TRUE. It also sends a
176 synchronization cookie by attaching it to {{SearchResultEntry}}
177 messages generated in the persist stage of the synchronization search. During
178 the persist stage, the provider can also send a Sync Info message
179 containing the synchronization cookie at any time the provider wants
180 to update the consumer-side state indicator.
182 In the LDAP Sync protocol, entries are uniquely identified by the
183 {{EX:entryUUID}} attribute value. It can function as a reliable
184 identifier of the entry. The DN of the entry, on the other hand,
185 can be changed over time and hence cannot be considered as the
186 reliable identifier. The {{EX:entryUUID}} is attached to each
187 {{SearchResultEntry}} or {{SearchResultReference}} as a part of the
188 synchronization control.
192 The syncrepl engine utilizes both the {{refreshOnly}} and the
193 {{refreshAndPersist}} operations of the LDAP Sync protocol. If a
194 syncrepl specification is included in a database definition,
195 {{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
196 and schedules its execution. If the {{refreshOnly}} operation is
197 specified, the syncrepl engine will be rescheduled at the interval
198 time after a synchronization operation is completed. If the
199 {{refreshAndPersist}} operation is specified, the engine will remain
200 active and process the persistent synchronization messages from the
203 The syncrepl engine utilizes both the present phase and the delete
204 phase of the refresh synchronization. It is possible to configure
205 a session log in the provider which stores the
206 {{EX:entryUUID}}s of a finite number of entries deleted from a
207 database. Multiple replicas share the same session log. The syncrepl
209 delete phase if the session log is present and the state of the
210 consumer server is recent enough that no session log entries are
211 truncated after the last synchronization of the client. The syncrepl
212 engine uses the present phase if no session log is configured for
213 the replication content or if the consumer replica is too outdated
214 to be covered by the session log. The current design of the session
215 log store is memory based, so the information contained in the
216 session log is not persistent over multiple provider invocations.
217 It is not currently supported to access the session log store by
218 using LDAP operations. It is also not currently supported to impose
219 access control to the session log.
221 As a further optimization, even in the case the synchronization
222 search is not associated with any session log, no entries will be
223 transmitted to the consumer server when there has been no update
224 in the replication context.
226 The syncrepl engine, which is a consumer-side replication engine,
227 can work with any backends. The LDAP Sync provider can be configured
228 as an overlay on any backend, but works best with the {{back-bdb}}
229 or {{back-hdb}} backend.
231 The LDAP Sync provider maintains a {{EX:contextCSN}} for each
232 database as the current synchronization state indicator of the
233 provider content. It is the largest {{EX:entryCSN}} in the provider
234 context such that no transactions for an entry having smaller
235 {{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}}
236 could not just be set to the largest issued {{EX:entryCSN}} because
237 {{EX:entryCSN}} is obtained before a transaction starts and
238 transactions are not committed in the issue order.
240 The provider stores the {{EX:contextCSN}} of a context in the
241 {{EX:contextCSN}} attribute of the context suffix entry. The attribute
242 is not written to the database after every update operation though;
243 instead it is maintained primarily in memory. At database start
244 time the provider reads the last saved {{EX:contextCSN}} into memory
245 and uses the in-memory copy exclusively thereafter. By default,
246 changes to the {{EX:contextCSN}} as a result of database updates
247 will not be written to the database until the server is cleanly
248 shut down. A checkpoint facility exists to cause the {{EX:contextCSN}} to
249 be written out more frequently if desired.
251 Note that at startup time, if the provider is unable to read a
252 {{EX:contextCSN}} from the suffix entry, it will scan the entire
253 database to determine the value, and this scan may take quite a
254 long time on a large database. When a {{EX:contextCSN}} value is
255 read, the database will still be scanned for any {{EX:entryCSN}}
256 values greater than it, to make sure the {{EX:contextCSN}} value
257 truly reflects the greatest committed {{EX:entryCSN}} in the database.
258 On databases which support inequality indexing, setting an eq index
259 on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
260 checkpoints will greatly speed up this scanning step.
262 If no {{EX:contextCSN}} can be determined by reading and scanning
263 the database, a new value will be generated. Also, if scanning the
264 database yielded a greater {{EX:entryCSN}} than was previously
265 recorded in the suffix entry's {{EX:contextCSN}} attribute, a
266 checkpoint will be immediately written with the new value.
268 The consumer also stores its replica state, which is the provider's
269 {{EX:contextCSN}} received as a synchronization cookie, in the
270 {{EX:contextCSN}} attribute of the suffix entry. The replica state
271 maintained by a consumer server is used as the synchronization state
272 indicator when it performs subsequent incremental synchronization
273 with the provider server. It is also used as a provider-side
274 synchronization state indicator when it functions as a secondary
275 provider server in a cascading replication configuration. Since
276 the consumer and provider state information are maintained in the
277 same location within their respective databases, any consumer can
278 be promoted to a provider (and vice versa) without any special
281 Because a general search filter can be used in the syncrepl
282 specification, some entries in the context may be omitted from the
283 synchronization content. The syncrepl engine creates a glue entry
284 to fill in the holes in the replica context if any part of the
285 replica content is subordinate to the holes. The glue entries will
286 not be returned in the search result unless {{ManageDsaIT}} control
289 Also as a consequence of the search filter used in the syncrepl
290 specification, it is possible for a modification to remove an entry
291 from the replication scope even though the entry has not been deleted
292 on the provider. Logically the entry must be deleted on the consumer
293 but in {{refreshOnly}} mode the provider cannot detect and propagate
294 this change without the use of the session log on the provider.
296 For configuration, please see the {{SECT:Syncrepl}} section.
299 H2: Deployment Alternatives
301 While the LDAP Sync specification only defines a narrow scope for replication,
302 the OpenLDAP implementation is extremely flexible and supports a variety of
303 operating modes to handle other scenarios not explicitly addressed in the spec.
306 H3: Delta-syncrepl replication
308 * Disadvantages of LDAP Sync replication:
310 LDAP Sync replication is an object-based replication mechanism.
311 When any attribute value in a replicated object is changed on the provider,
312 each consumer fetches and processes the complete changed object, including
313 {{B:both the changed and unchanged attribute values}} during replication.
314 One advantage of this approach is that when multiple changes occur to
315 a single object, the precise sequence of those changes need not be preserved;
316 only the final state of the entry is significant. But this approach
317 may have drawbacks when the usage pattern involves single changes to
320 For example, suppose you have a database consisting of 100,000 objects of 1 KB
321 each. Further, suppose you routinely run a batch job to change the value of
322 a single two-byte attribute value that appears in each of the 100,000 objects
323 on the master. Not counting LDAP and TCP/IP protocol overhead, each time you
324 run this job each consumer will transfer and process {{B:1 GB}} of data to
325 process {{B:200KB of changes!}}
327 99.98% of the data that is transmitted and processed in a case like this will
328 be redundant, since it represents values that did not change. This is a waste
329 of valuable transmission and processing bandwidth and can cause an unacceptable
330 replication backlog to develop. While this situation is extreme, it serves to
331 demonstrate a very real problem that is encountered in some LDAP deployments.
334 * Where Delta-syncrepl comes in:
336 Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address
337 situations like the one described above. Delta-syncrepl works by maintaining a
338 changelog of a selectable depth on the provider. The replication consumer
339 checks the changelog for the changes it needs and, as long as
340 the changelog contains the needed changes, the consumer fetches the changes
341 from the changelog and applies them to its database. If, however, a replica
342 is too far out of sync (or completely empty), conventional syncrepl is used to
343 bring it up to date and replication then switches back to the delta-syncrepl
346 For configuration, please see the {{SECT:Delta-syncrepl}} section.
349 H3: N-Way Multi-Master replication
351 Multi-Master replication is a replication technique using Syncrepl to replicate
352 data to multiple provider ("Master") Directory servers.
354 H4: Valid Arguments for Multi-Master replication
356 * If any provider fails, other providers will continue to accept updates
357 * Avoids a single point of failure
358 * Providers can be located in several physical sites i.e. distributed across
360 * Good for Automatic failover/High Availability
362 H4: Invalid Arguments for Multi-Master replication
364 (These are often claimed to be advantages of Multi-Master replication but
365 those claims are false):
367 * It has {{B:NOTHING}} to do with load balancing
368 * Providers {{B:must}} propagate writes to {{B:all}} the other servers, which
369 means the network traffic and write load spreads across all
370 of the servers the same as for single-master.
371 * Server utilization and performance are at best identical for
372 Multi-Master and Single-Master replication; at worst Single-Master is
373 superior because indexing can be tuned differently to optimize for the
374 different usage patterns between the provider and the consumers.
376 H4: Arguments against Multi-Master replication
378 * Breaks the data consistency guarantees of the directory model
379 * {{URL:http://www.openldap.org/faq/data/cache/1240.html}}
380 * If connectivity with a provider is lost because of a network partition, then
381 "automatic failover" can just compound the problem
382 * Typically, a particular machine cannot distinguish between losing contact
383 with a peer because that peer crashed, or because the network link has failed
384 * If a network is partitioned and multiple clients start writing to each of the
385 "masters" then reconciliation will be a pain; it may be best to simply deny
386 writes to the clients that are partitioned from the single provider
389 For configuration, please see the {{SECT:N-Way Multi-Master}} section below
391 H3: MirrorMode replication
393 MirrorMode is a hybrid configuration that provides all of the consistency
394 guarantees of single-master replication, while also providing the high
395 availability of multi-master. In MirrorMode two providers are set up to
396 replicate from each other (as a multi-master configuration), but an
397 external frontend is employed to direct all writes to only one of
398 the two servers. The second provider will only be used for writes if
399 the first provider crashes, at which point the frontend will switch to
400 directing all writes to the second provider. When a crashed provider is
401 repaired and restarted it will automatically catch up to any changes
402 on the running provider and resync.
404 H4: Arguments for MirrorMode
406 * Provides a high-availability (HA) solution for directory writes (replicas handle reads)
407 * As long as one provider is operational, writes can safely be accepted
408 * Provider nodes replicate from each other, so they are always up to date and
409 can be ready to take over (hot standby)
410 * Syncrepl also allows the provider nodes to re-synchronize after any downtime
413 H4: Arguments against MirrorMode
415 * MirrorMode is not what is termed as a Multi-Master solution. This is because
416 writes have to go to just one of the mirror nodes at a time
417 * MirrorMode can be termed as Active-Active Hot-Standby, therefore an external
418 server (slapd in proxy mode) or device (hardware load balancer)
419 is needed to manage which provider is currently active
420 * Backups are managed slightly differently
421 - If backing up the Berkeley database itself and periodically backing up the
422 transaction log files, then the same member of the mirror pair needs to be
423 used to collect logfiles until the next database backup is taken
424 - To ensure that both databases are consistent, each database might have to be
425 put in read-only mode while performing a slapcat.
426 * Delta-Syncrepl is not yet supported
428 For configuration, please see the {{SECT:MirrorMode}} section below
431 H3: Syncrepl Proxy Mode
433 While the LDAP Sync protocol supports both pull- and push-based replication,
434 the push mode (refreshAndPersist) must still be initiated from the consumer
435 before the provider can begin pushing changes. In some network configurations,
436 particularly where firewalls restrict the direction in which connections
437 can be made, a provider-initiated push mode may be needed.
439 This mode can be configured with the aid of the LDAP Backend
440 ({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the
441 syncrepl engine on the actual consumer, a slapd-ldap proxy is set up
442 near (or collocated with) the provider that points to the consumer,
443 and the syncrepl engine runs on the proxy.
445 For configuration, please see the {{SECT:Syncrepl Proxy}} section.
449 The old {{slurpd}} mechanism only operated in provider-initiated
450 push mode. Slurpd replication was deprecated in favor of Syncrepl
451 replication and has been completely removed from OpenLDAP 2.4.
453 The slurpd daemon was the original replication mechanism inherited from
454 UMich's LDAP and operated in push mode: the master pushed changes to the
455 slaves. It was replaced for many reasons, in brief:
457 * It was not reliable
458 ** It was extremely sensitive to the ordering of records in the replog
459 ** It could easily go out of sync, at which point manual intervention was
460 required to resync the slave database with the master directory
461 ** It wasn't very tolerant of unavailable servers. If a slave went down
462 for a long time, the replog could grow to a size that was too large for
464 * It only worked in push mode
465 * It required stopping and restarting the master to add new slaves
466 * It only supported single master replication
468 Syncrepl has none of those weaknesses:
470 * Syncrepl is self-synchronizing; you can start with a consumer database
471 in any state from totally empty to fully synced and it will automatically
472 do the right thing to achieve and maintain synchronization
473 ** It is completely insensitive to the order in which changes occur
474 ** It guarantees convergence between the consumer and the provider
475 content without manual intervention
476 ** It can resynchronize regardless of how long a consumer stays out
477 of contact with the provider
478 * Syncrepl can operate in either direction
479 * Consumers can be added at any time without touching anything on the
481 * Multi-master replication is supported
484 H2: Configuring the different replication types
488 H4: Syncrepl configuration
490 Because syncrepl is a consumer-side replication engine, the syncrepl
491 specification is defined in {{slapd.conf}}(5) of the consumer
492 server, not in the provider server's configuration file. The initial
493 loading of the replica content can be performed either by starting
494 the syncrepl engine with no synchronization cookie or by populating
495 the consumer replica by loading an {{TERM:LDIF}} file dumped as a
496 backup at the provider.
498 When loading from a backup, it is not required to perform the initial
499 loading from the up-to-date backup of the provider content. The
500 syncrepl engine will automatically synchronize the initial consumer
501 replica to the current provider content. As a result, it is not
502 required to stop the provider server in order to avoid the replica
503 inconsistency caused by the updates to the provider content during
504 the content backup and loading process.
506 When replicating a large scale directory, especially in a bandwidth
507 constrained environment, it is advised to load the consumer replica
508 from a backup instead of performing a full initial load using
512 H4: Set up the provider slapd
514 The provider is implemented as an overlay, so the overlay itself
515 must first be configured in {{slapd.conf}}(5) before it can be
516 used. The provider has only two configuration directives, for setting
517 checkpoints on the {{EX:contextCSN}} and for configuring the session
518 log. Because the LDAP Sync search is subject to access control,
519 proper access control privileges should be set up for the replicated
522 The {{EX:contextCSN}} checkpoint is configured by the
524 > syncprov-checkpoint <ops> <minutes>
526 directive. Checkpoints are only tested after successful write
527 operations. If {{<ops>}} operations or more than {{<minutes>}}
528 time has passed since the last checkpoint, a new checkpoint is
531 The session log is configured by the
533 > syncprov-sessionlog <size>
535 directive, where {{<size>}} is the maximum number of session log
536 entries the session log can record. When a session log is configured,
537 it is automatically used for all LDAP Sync searches within the
540 Note that using the session log requires searching on the {{entryUUID}}
541 attribute. Setting an eq index on this attribute will greatly benefit
542 the performance of the session log on the provider.
544 A more complete example of the {{slapd.conf}}(5) content is thus:
547 > suffix dc=Example,dc=com
548 > rootdn dc=Example,dc=com
549 > directory /var/ldap/db
550 > index objectclass,entryCSN,entryUUID eq
553 > syncprov-checkpoint 100 10
554 > syncprov-sessionlog 100
557 H4: Set up the consumer slapd
559 The syncrepl replication is specified in the database section of
560 {{slapd.conf}}(5) for the replica context. The syncrepl engine
561 is backend independent and the directive can be defined with any
565 > suffix dc=Example,dc=com
566 > rootdn dc=Example,dc=com
567 > directory /var/ldap/db
568 > index objectclass,entryCSN,entryUUID eq
571 > provider=ldap://provider.example.com:389
573 > interval=01:00:00:00
574 > searchbase="dc=example,dc=com"
575 > filter="(objectClass=organizationalPerson)"
577 > attrs="cn,sn,ou,telephoneNumber,title,l"
580 > binddn="cn=syncuser,dc=example,dc=com"
583 In this example, the consumer will connect to the provider {{slapd}}(8)
584 at port 389 of {{FILE:ldap://provider.example.com}} to perform a
585 polling ({{refreshOnly}}) mode of synchronization once a day. It
586 will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
587 authentication with password "secret". Note that the access control
588 privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
589 appropriately in the provider to retrieve the desired replication
590 content. Also the search limits must be high enough on the provider
591 to allow the syncuser to retrieve a complete copy of the requested
592 content. The consumer uses the rootdn to write to its database so
593 it always has full permissions to write all content.
595 The synchronization search in the above example will search for the
596 entries whose objectClass is organizationalPerson in the entire
597 subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
598 are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
599 {{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
600 that the consumer {{slapd}}(8) will not enforce entry schema
601 checking when it processes updates from the provider {{slapd}}(8).
603 For more detailed information on the syncrepl directive, see the
604 {{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
605 chapter of this admin guide.
608 H4: Start the provider and the consumer slapd
610 The provider {{slapd}}(8) is not required to be restarted.
611 {{contextCSN}} is automatically generated as needed: it might be
612 originally contained in the {{TERM:LDIF}} file, generated by
613 {{slapadd}} (8), generated upon changes in the context, or generated
614 when the first LDAP Sync search arrives at the provider. If an
615 LDIF file is being loaded which did not previously contain the
616 {{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
617 (8) to cause it to be generated. This will allow the server to
618 startup a little quicker the first time it runs.
620 When starting a consumer {{slapd}}(8), it is possible to provide
621 a synchronization cookie as the {{-c cookie}} command line option
622 in order to start the synchronization from a specific state. The
623 cookie is a comma separated list of name=value pairs. Currently
624 supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
625 {{<csn>}} represents the current synchronization state of the
626 consumer replica. {{<rid>}} identifies a consumer replica locally
627 within the consumer server. It is used to relate the cookie to the
628 syncrepl definition in {{slapd.conf}}(5) which has the matching
629 replica identifier. The {{<rid>}} must have no more than 3 decimal
630 digits. The command line cookie overrides the synchronization
631 cookie stored in the consumer replica database.
636 H4: Delta-syncrepl Provider configuration
638 Setting up delta-syncrepl requires configuration changes on both the master and
641 > # Give the replica DN unlimited read access. This ACL needs to be
642 > # merged with other ACL statements, and/or moved within the scope
643 > # of a database. The "by * break" portion causes evaluation of
644 > # subsequent rules. See slapd.access(5) for details.
646 > by dn.base="cn=replicator,dc=symas,dc=com" read
649 > # Set the module path location
650 > modulepath /opt/symas/lib/openldap
652 > # Load the hdb backend
653 > moduleload back_hdb.la
655 > # Load the accesslog overlay
656 > moduleload accesslog.la
658 > #Load the syncprov overlay
659 > moduleload syncprov.la
661 > # Accesslog database definitions
663 > suffix cn=accesslog
664 > directory /db/accesslog
665 > rootdn cn=accesslog
667 > index entryCSN,objectClass,reqEnd,reqResult,reqStart
670 > syncprov-nopresent TRUE
671 > syncprov-reloadhint TRUE
673 > # Let the replica DN have limitless searches
674 > limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
676 > # Primary database definitions
678 > suffix "dc=symas,dc=com"
679 > rootdn "cn=manager,dc=symas,dc=com"
681 > ## Whatever other configuration options are desired
683 > # syncprov specific indexing
687 > # syncrepl Provider for primary db
689 > syncprov-checkpoint 1000 60
691 > # accesslog overlay definitions for primary db
696 > # scan the accesslog DB every day, and purge entries older than 7 days
697 > logpurge 07+00:00 01+00:00
699 > # Let the replica DN have limitless searches
700 > limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
702 For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5))
705 H4: Delta-syncrepl Consumer configuration
707 > # Replica database configuration
709 > suffix "dc=symas,dc=com"
710 > rootdn "cn=manager,dc=symas,dc=com"
712 > ## Whatever other configuration bits for the replica, like indexing
715 > # syncrepl specific indices
718 > # syncrepl directives
720 > provider=ldap://ldapmaster.symas.com:389
722 > binddn="cn=replicator,dc=symas,dc=com"
724 > searchbase="dc=symas,dc=com"
725 > logbase="cn=accesslog"
726 > logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
728 > type=refreshAndPersist
732 > # Refer updates to the master
733 > updateref ldap://ldapmaster.symas.com
736 The above configuration assumes that you have a replicator identity defined
737 in your database that can be used to bind to the provider. In addition,
738 all of the databases (primary, replica, and the accesslog
739 storage database) should also have properly tuned {{DB_CONFIG}} files that meet
743 H3: N-Way Multi-Master
745 For the following example we will be using 3 Master nodes. Keeping in line with
746 {{B:test050-syncrepl-multimaster}} of the OpenLDAP test suite, we will be configuring
747 {{slapd(8)}} via {{B:cn=config}}
749 This sets up the config database:
752 > objectClass: olcGlobal
756 > dn: olcDatabase={0}config,cn=config
757 > objectClass: olcDatabaseConfig
758 > olcDatabase: {0}config
761 second and third servers will have a different olcServerID obviously:
764 > objectClass: olcGlobal
768 > dn: olcDatabase={0}config,cn=config
769 > objectClass: olcDatabaseConfig
770 > olcDatabase: {0}config
773 This sets up syncrepl as a provider (since these are all masters):
775 > dn: cn=module,cn=config
776 > objectClass: olcModuleList
778 > olcModulePath: /usr/local/libexec/openldap
779 > olcModuleLoad: syncprov.la
781 Now we setup the first Master Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls):
785 > replace: olcServerID
786 > olcServerID: 1 $URI1
787 > olcServerID: 2 $URI2
788 > olcServerID: 3 $URI3
790 > dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
792 > objectClass: olcOverlayConfig
793 > objectClass: olcSyncProvConfig
794 > olcOverlay: syncprov
796 > dn: olcDatabase={0}config,cn=config
799 > olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
800 > credentials=secret searchbase="cn=config" type=refreshAndPersist
801 > retry="5 5 300 5" timeout=1
802 > olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
803 > credentials=secret searchbase="cn=config" type=refreshAndPersist
804 > retry="5 5 300 5" timeout=1
805 > olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
806 > credentials=secret searchbase="cn=config" type=refreshAndPersist
807 > retry="5 5 300 5" timeout=1
810 > olcMirrorMode: TRUE
812 Now start up the Master and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multimaster on the config database.
814 We still have to replicate the actual data, not just the config, so add to the master (all active and configured consumers/masters will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup:
816 > dn: olcDatabase={1}$BACKEND,cn=config
817 > objectClass: olcDatabaseConfig
818 > objectClass: olc${BACKEND}Config
819 > olcDatabase: {1}$BACKEND
821 > olcDbDirectory: ./db
822 > olcRootDN: $MANAGERDN
824 > olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
825 > olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple
826 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
827 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
828 > olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple
829 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
830 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
831 > olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple
832 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
833 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
834 > olcMirrorMode: TRUE
836 > dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config
838 > objectClass: olcOverlayConfig
839 > objectClass: olcSyncProvConfig
840 > olcOverlay: syncprov
842 Note: All of your servers' clocks must be tightly synchronized using
843 e.g. NTP {{http://www.ntp.org/}}, atomic clock, or some other reliable
846 Note: As stated in {{slapd-config}}(5), URLs specified in {{olcSyncRepl}}
847 directives are the URLs of the servers from which to replicate. These
848 must exactly match the URLs {{slapd}} listens on ({{-h}} in {{SECT:Command-Line Options}}).
849 Otherwise slapd may attempt to replicate from itself, causing a loop.
853 MirrorMode configuration is actually very easy. If you have ever setup a normal
854 slapd syncrepl provider, then the only change is the following two directives:
859 Note: You need to make sure that the {{serverID}} of each mirror node is
860 different and add it as a global configuration option.
862 H4: Mirror Node Configuration
864 The first step is to configure the syncrepl provider the same as in the
865 {{SECT:Set up the provider slapd}} section.
867 Note: Delta-syncrepl is not yet supported with MirrorMode.
869 Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in
870 {{refreshAndPersist}} mode:
878 > # syncrepl directive
\r
880 > provider=ldap://ldap-sid2.example.com
\r
881 > bindmethod=simple
\r
882 > binddn="cn=mirrormode,dc=example,dc=com"
\r
883 > credentials=mirrormode
\r
884 > searchbase="dc=example,dc=com"
\r
885 > schemachecking=on
\r
886 > type=refreshAndPersist
\r
897 > # syncrepl directive
\r
899 > provider=ldap://ldap-sid1.example.com
\r
900 > bindmethod=simple
\r
901 > binddn="cn=mirrormode,dc=example,dc=com"
\r
902 > credentials=mirrormode
\r
903 > searchbase="dc=example,dc=com"
\r
904 > schemachecking=on
\r
905 > type=refreshAndPersist
\r
910 It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except
911 that the {{serverID}} is unique, and each consumer is pointed to
914 H5: Failover Configuration
916 There are generally 2 choices for this; 1. Hardware proxies/load-balancing or
917 dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider
919 A typical enterprise example might be:
921 !import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration"
922 FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration
924 H5: Normal Consumer Configuration
926 This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It
927 can either setup in normal {{SECT:syncrepl replication}} mode, or in
928 {{SECT:delta-syncrepl replication}} mode.
930 H4: MirrorMode Summary
932 You will now have a directory architecture that provides all of the
933 consistency guarantees of single-master replication, while also providing the
934 high availability of multi-master replication.
939 !import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode"
940 FT[align="Center"] Figure X.Y: Replacing slurpd
942 The following example is for a self-contained push-based replication solution:
944 > #######################################################################
945 > # Standard OpenLDAP Master/Provider
946 > #######################################################################
948 > include /usr/local/etc/openldap/schema/core.schema
949 > include /usr/local/etc/openldap/schema/cosine.schema
950 > include /usr/local/etc/openldap/schema/nis.schema
951 > include /usr/local/etc/openldap/schema/inetorgperson.schema
953 > include /usr/local/etc/openldap/slapd.acl
955 > modulepath /usr/local/libexec/openldap
956 > moduleload back_hdb.la
957 > moduleload syncprov.la
958 > moduleload back_monitor.la
959 > moduleload back_ldap.la
961 > pidfile /usr/local/var/slapd.pid
962 > argsfile /usr/local/var/slapd.args
964 > loglevel sync stats
967 > suffix "dc=suretecsystems,dc=com"
968 > directory /usr/local/var/openldap-data
974 > index objectClass eq
978 > rootdn "cn=admin,dc=suretecsystems,dc=com"
981 > # syncprov specific indexing
985 > # syncrepl Provider for primary db
987 > syncprov-checkpoint 1000 60
989 > # Let the replica DN have limitless searches
990 > limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
997 > ##############################################################################
998 > # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
999 > ##############################################################################
1002 > # ignore conflicts with other databases, as we need to push out to same suffix
1004 > suffix "dc=suretecsystems,dc=com"
1005 > rootdn "cn=slapd-ldap"
1006 > uri ldap://localhost:9012/
1010 > # We don't need any access to this DSA
1013 > acl-bind bindmethod=simple
1014 > binddn="cn=replicator,dc=suretecsystems,dc=com"
1015 > credentials=testing
1018 > provider=ldap://localhost:9011/
1019 > binddn="cn=replicator,dc=suretecsystems,dc=com"
1021 > credentials=testing
1022 > searchbase="dc=suretecsystems,dc=com"
1023 > type=refreshAndPersist
1028 A replica configuration for this type of setup could be:
1030 > #######################################################################
1031 > # Standard OpenLDAP Slave without Syncrepl
1032 > #######################################################################
1034 > include /usr/local/etc/openldap/schema/core.schema
1035 > include /usr/local/etc/openldap/schema/cosine.schema
1036 > include /usr/local/etc/openldap/schema/nis.schema
1037 > include /usr/local/etc/openldap/schema/inetorgperson.schema
1039 > include /usr/local/etc/openldap/slapd.acl
1041 > modulepath /usr/local/libexec/openldap
1042 > moduleload back_hdb.la
1043 > moduleload syncprov.la
1044 > moduleload back_monitor.la
1045 > moduleload back_ldap.la
1047 > pidfile /usr/local/var/slapd.pid
1048 > argsfile /usr/local/var/slapd.args
1050 > loglevel sync stats
1053 > suffix "dc=suretecsystems,dc=com"
1054 > directory /usr/local/var/openldap-slave/data
1058 > idlcachesize 10000
1060 > index objectClass eq
1064 > rootdn "cn=admin,dc=suretecsystems,dc=com"
1067 > # Let the replica DN have limitless searches
1068 > limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
1070 > updatedn "cn=replicator,dc=suretecsystems,dc=com"
1072 > # Refer updates to the master
1073 > updateref ldap://localhost:9011
1080 You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be:
1082 > # Give the replica DN unlimited read access. This ACL may need to be
1083 > # merged with other ACL statements.
1086 > by dn.base="cn=replicator,dc=suretecsystems,dc=com" write
1089 > access to dn.base=""
1092 > access to dn.base="cn=Subschema"
1095 > access to dn.subtree="cn=Monitor"
1096 > by dn.exact="uid=admin,dc=suretecsystems,dc=com" write
1104 In order to support more replicas, just add more {{database ldap}} sections and
1105 increment the {{syncrepl rid}} number accordingly.
1107 Note: You must populate the Master and Slave directories with the same data,
1108 unlike when using normal Syncrepl
1110 If you do not have access to modify the master directory configuration you can
1111 configure a standalone ldap proxy, which might look like:
1113 !import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode"
1114 FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version
1116 The following configuration is an example of a standalone LDAP Proxy:
1118 > include /usr/local/etc/openldap/schema/core.schema
1119 > include /usr/local/etc/openldap/schema/cosine.schema
1120 > include /usr/local/etc/openldap/schema/nis.schema
1121 > include /usr/local/etc/openldap/schema/inetorgperson.schema
1123 > include /usr/local/etc/openldap/slapd.acl
1125 > modulepath /usr/local/libexec/openldap
1126 > moduleload syncprov.la
1127 > moduleload back_ldap.la
1129 > ##############################################################################
1130 > # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
1131 > ##############################################################################
1134 > # ignore conflicts with other databases, as we need to push out to same suffix
1136 > suffix "dc=suretecsystems,dc=com"
1137 > rootdn "cn=slapd-ldap"
1138 > uri ldap://localhost:9012/
1142 > # We don't need any access to this DSA
1145 > acl-bind bindmethod=simple
1146 > binddn="cn=replicator,dc=suretecsystems,dc=com"
1147 > credentials=testing
1150 > provider=ldap://localhost:9011/
1151 > binddn="cn=replicator,dc=suretecsystems,dc=com"
1153 > credentials=testing
1154 > searchbase="dc=suretecsystems,dc=com"
1155 > type=refreshAndPersist
1160 As you can see, you can let your imagination go wild using Syncrepl and
1161 {{slapd-ldap(8)}} tailoring your replication to fit your specific network