2 # Copyright 1999-2008 The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
7 Replicated directories are a fundamental requirement for delivering a
8 resilient enterprise deployment.
10 {{PRD:OpenLDAP}} has various configuration options for creating a replicated
11 directory. The following sections will discuss these.
18 {{Slurpd}} replication has been deprecated in favor of Syncrepl replication and
19 has been completely removed from OpenLDAP 2.4.
21 {{Why was it replaced?}}
23 The {{slurpd}} daemon was the original replication mechanism inherited from
24 UMich's LDAP and operates in push mode: the master pushes changes to the
25 slaves. It has been replaced for many reasons, in brief:
28 * It is extremely sensitive to the ordering of records in the replog
29 * It can easily go out of sync, at which point manual intervention is
30 required to resync the slave database with the master directory
31 * It isn't very tolerant of unavailable servers. If a slave goes down
32 for a long time, the replog may grow to a size that's too large for
35 {{What was it replaced with?}}
39 {{Why is Syncrepl better?}}
41 * Syncrepl is self-synchronizing; you can start with a database in any
42 state from totally empty to fully synced and it will automatically do
43 the right thing to achieve and maintain synchronization
44 * Syncrepl can operate in either direction
45 * Data updates can be minimal or maximal
47 {{How do I implement a pushed based replication system using Syncrepl?}}
49 The easiest way is to point an LDAP backend ({{SECT: Backends}} and {{slapd-ldap(8)}})
50 to your slave directory and setup Syncrepl to point to your Master database.
52 If you imagine Syncrepl pulling down changes from the Master server, and then
53 pushing those changes out to your slave servers via {{slapd-ldap(8)}}. This is
54 called Syncrepl Proxy Mode. You can also use Syncrepl Multi-proxy mode:
56 !import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode"
57 FT[align="Center"] Figure X.Y: Replacing slurpd
59 The following example is for a self-contained push-based replication solution:
61 > #######################################################################
62 > # Standard OpenLDAP Master/Provider
63 > #######################################################################
65 > include /usr/local/etc/openldap/schema/core.schema
66 > include /usr/local/etc/openldap/schema/cosine.schema
67 > include /usr/local/etc/openldap/schema/nis.schema
68 > include /usr/local/etc/openldap/schema/inetorgperson.schema
70 > include /usr/local/etc/openldap/slapd.acl
72 > modulepath /usr/local/libexec/openldap
73 > moduleload back_hdb.la
74 > moduleload syncprov.la
75 > moduleload back_monitor.la
76 > moduleload back_ldap.la
78 > pidfile /usr/local/var/slapd.pid
79 > argsfile /usr/local/var/slapd.args
84 > suffix "dc=suretecsystems,dc=com"
85 > directory /usr/local/var/openldap-data
91 > index objectClass eq
95 > rootdn "cn=admin,dc=suretecsystems,dc=com"
98 > # syncprov specific indexing
102 > # syncrepl Provider for primary db
104 > syncprov-checkpoint 1000 60
106 > # Let the replica DN have limitless searches
107 > limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
114 > ##############################################################################
115 > # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
116 > ##############################################################################
119 > # ignore conflicts with other databases, as we need to push out to same suffix
121 > suffix "dc=suretecsystems,dc=com"
122 > rootdn "cn=slapd-ldap"
123 > uri ldap://localhost:9012/
127 > # We don't need any access to this DSA
130 > acl-bind bindmethod=simple
131 > binddn="cn=replicator,dc=suretecsystems,dc=com"
132 > credentials=testing
135 > provider=ldap://localhost:9011/
136 > binddn="cn=replicator,dc=suretecsystems,dc=com"
138 > credentials=testing
139 > searchbase="dc=suretecsystems,dc=com"
140 > type=refreshAndPersist
145 A replica configuration for this type of setup could be:
147 > #######################################################################
148 > # Standard OpenLDAP Slave without Syncrepl
149 > #######################################################################
151 > include /usr/local/etc/openldap/schema/core.schema
152 > include /usr/local/etc/openldap/schema/cosine.schema
153 > include /usr/local/etc/openldap/schema/nis.schema
154 > include /usr/local/etc/openldap/schema/inetorgperson.schema
156 > include /usr/local/etc/openldap/slapd.acl
158 > modulepath /usr/local/libexec/openldap
159 > moduleload back_hdb.la
160 > moduleload syncprov.la
161 > moduleload back_monitor.la
162 > moduleload back_ldap.la
164 > pidfile /usr/local/var/slapd.pid
165 > argsfile /usr/local/var/slapd.args
167 > loglevel sync stats
170 > suffix "dc=suretecsystems,dc=com"
171 > directory /usr/local/var/openldap-slave/data
177 > index objectClass eq
181 > rootdn "cn=admin,dc=suretecsystems,dc=com"
184 > # Let the replica DN have limitless searches
185 > limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
187 > updatedn "cn=replicator,dc=suretecsystems,dc=com"
189 > # Refer updates to the master
190 > updateref ldap://localhost:9011
197 You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be:
199 > # Give the replica DN unlimited read access. This ACL may need to be
200 > # merged with other ACL statements.
203 > by dn.base="cn=replicator,dc=suretecsystems,dc=com" write
206 > access to dn.base=""
209 > access to dn.base="cn=Subschema"
212 > access to dn.subtree="cn=Monitor"
213 > by dn.exact="uid=admin,dc=suretecsystems,dc=com" write
221 In order to support more replicas, just add more {{database ldap}} sections and
222 increment the {{syncrepl rid}} number accordingly.
224 Note: You must populate the Master and Slave directories with the same data,
225 unlike when using normal Syncrepl
227 If you do not have access to modify the master directory configuration you can
228 configure a standalone ldap proxy, which might look like:
230 !import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode"
231 FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version
233 The following configuration is an example of a standalone LDAP Proxy:
235 > include /usr/local/etc/openldap/schema/core.schema
236 > include /usr/local/etc/openldap/schema/cosine.schema
237 > include /usr/local/etc/openldap/schema/nis.schema
238 > include /usr/local/etc/openldap/schema/inetorgperson.schema
240 > include /usr/local/etc/openldap/slapd.acl
242 > modulepath /usr/local/libexec/openldap
243 > moduleload syncprov.la
244 > moduleload back_ldap.la
246 > ##############################################################################
247 > # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
248 > ##############################################################################
251 > # ignore conflicts with other databases, as we need to push out to same suffix
253 > suffix "dc=suretecsystems,dc=com"
254 > rootdn "cn=slapd-ldap"
255 > uri ldap://localhost:9012/
259 > # We don't need any access to this DSA
262 > acl-bind bindmethod=simple
263 > binddn="cn=replicator,dc=suretecsystems,dc=com"
264 > credentials=testing
267 > provider=ldap://localhost:9011/
268 > binddn="cn=replicator,dc=suretecsystems,dc=com"
270 > credentials=testing
271 > searchbase="dc=suretecsystems,dc=com"
272 > type=refreshAndPersist
277 As you can see, you can let your imagination go wild using Syncrepl and
278 {{slapd-ldap(8)}} tailoring your replication to fit your specific network
283 H3: LDAP Sync Replication
285 The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
286 short, is a consumer-side replication engine that enables the
287 consumer {{TERM:LDAP}} server to maintain a shadow copy of a
288 {{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
289 as one of the {{slapd}}(8) threads. It creates and maintains a
290 consumer replica by connecting to the replication provider to perform
291 the initial DIT content load followed either by periodic content
292 polling or by timely updates upon content changes.
294 Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
295 short) protocol as the replica synchronization protocol. It provides
296 a stateful replication which supports both pull-based and push-based
297 synchronization and does not mandate the use of a history store.
299 Syncrepl keeps track of the status of the replication content by
300 maintaining and exchanging synchronization cookies. Because the
301 syncrepl consumer and provider maintain their content status, the
302 consumer can poll the provider content to perform incremental
303 synchronization by asking for the entries required to make the
304 consumer replica up-to-date with the provider content. Syncrepl
305 also enables convenient management of replicas by maintaining replica
306 status. The consumer replica can be constructed from a consumer-side
307 or a provider-side backup at any synchronization status. Syncrepl
308 can automatically resynchronize the consumer replica up-to-date
309 with the current provider content.
311 Syncrepl supports both pull-based and push-based synchronization.
312 In its basic refreshOnly synchronization mode, the provider uses
313 pull-based synchronization where the consumer servers need not be
314 tracked and no history information is maintained. The information
315 required for the provider to process periodic polling requests is
316 contained in the synchronization cookie of the request itself. To
317 optimize the pull-based synchronization, syncrepl utilizes the
318 present phase of the LDAP Sync protocol as well as its delete phase,
319 instead of falling back on frequent full reloads. To further optimize
320 the pull-based synchronization, the provider can maintain a per-scope
321 session log as a history store. In its refreshAndPersist mode of
322 synchronization, the provider uses a push-based synchronization.
323 The provider keeps track of the consumer servers that have requested
324 a persistent search and sends them necessary updates as the provider
325 replication content gets modified.
327 With syncrepl, a consumer server can create a replica without
328 changing the provider's configurations and without restarting the
329 provider server, if the consumer server has appropriate access
330 privileges for the DIT fragment to be replicated. The consumer
331 server can stop the replication also without the need for provider-side
334 Syncrepl supports both partial and sparse replications. The shadow
335 DIT fragment is defined by a general search criteria consisting of
336 base, scope, filter, and attribute list. The replica content is
337 also subject to the access privileges of the bind identity of the
338 syncrepl replication connection.
341 H4: The LDAP Content Synchronization Protocol
343 The LDAP Sync protocol allows a client to maintain a synchronized
344 copy of a DIT fragment. The LDAP Sync operation is defined as a set
345 of controls and other protocol elements which extend the LDAP search
346 operation. This section introduces the LDAP Content Sync protocol
347 only briefly. For more information, refer to {{REF:RFC4533}}.
349 The LDAP Sync protocol supports both polling and listening for
350 changes by defining two respective synchronization operations:
351 {{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
352 by the {{refreshOnly}} operation. The client copy is synchronized
353 to the server copy at the time of polling. The server finishes the
354 search operation by returning {{SearchResultDone}} at the end of
355 the search operation as in the normal search. The listening is
356 implemented by the {{refreshAndPersist}} operation. Instead of
357 finishing the search after returning all entries currently matching
358 the search criteria, the synchronization search remains persistent
359 in the server. Subsequent updates to the synchronization content
360 in the server cause additional entry updates to be sent to the
363 The {{refreshOnly}} operation and the refresh stage of the
364 {{refreshAndPersist}} operation can be performed with a present
365 phase or a delete phase.
367 In the present phase, the server sends the client the entries updated
368 within the search scope since the last synchronization. The server
369 sends all requested attributes, be it changed or not, of the updated
370 entries. For each unchanged entry which remains in the scope, the
371 server sends a present message consisting only of the name of the
372 entry and the synchronization control representing state present.
373 The present message does not contain any attributes of the entry.
374 After the client receives all update and present entries, it can
375 reliably determine the new client copy by adding the entries added
376 to the server, by replacing the entries modified at the server, and
377 by deleting entries in the client copy which have not been updated
378 nor specified as being present at the server.
380 The transmission of the updated entries in the delete phase is the
381 same as in the present phase. The server sends all the requested
382 attributes of the entries updated within the search scope since the
383 last synchronization to the client. In the delete phase, however,
384 the server sends a delete message for each entry deleted from the
385 search scope, instead of sending present messages. The delete
386 message consists only of the name of the entry and the synchronization
387 control representing state delete. The new client copy can be
388 determined by adding, modifying, and removing entries according to
389 the synchronization control attached to the {{SearchResultEntry}}
392 In the case that the LDAP Sync server maintains a history store and
393 can determine which entries are scoped out of the client copy since
394 the last synchronization time, the server can use the delete phase.
395 If the server does not maintain any history store, cannot determine
396 the scoped-out entries from the history store, or the history store
397 does not cover the outdated synchronization state of the client,
398 the server should use the present phase. The use of the present
399 phase is much more efficient than a full content reload in terms
400 of the synchronization traffic. To reduce the synchronization
401 traffic further, the LDAP Sync protocol also provides several
402 optimizations such as the transmission of the normalized {{EX:entryUUID}}s
403 and the transmission of multiple {{EX:entryUUIDs}} in a single
404 {{syncIdSet}} message.
406 At the end of the {{refreshOnly}} synchronization, the server sends
407 a synchronization cookie to the client as a state indicator of the
408 client copy after the synchronization is completed. The client
409 will present the received cookie when it requests the next incremental
410 synchronization to the server.
412 When {{refreshAndPersist}} synchronization is used, the server sends
413 a synchronization cookie at the end of the refresh stage by sending
414 a Sync Info message with TRUE refreshDone. It also sends a
415 synchronization cookie by attaching it to {{SearchResultEntry}}
416 generated in the persist stage of the synchronization search. During
417 the persist stage, the server can also send a Sync Info message
418 containing the synchronization cookie at any time the server wants
419 to update the client-side state indicator. The server also updates
420 a synchronization indicator of the client at the end of the persist
423 In the LDAP Sync protocol, entries are uniquely identified by the
424 {{EX:entryUUID}} attribute value. It can function as a reliable
425 identifier of the entry. The DN of the entry, on the other hand,
426 can be changed over time and hence cannot be considered as the
427 reliable identifier. The {{EX:entryUUID}} is attached to each
428 {{SearchResultEntry}} or {{SearchResultReference}} as a part of the
429 synchronization control.
434 The syncrepl engine utilizes both the {{refreshOnly}} and the
435 {{refreshAndPersist}} operations of the LDAP Sync protocol. If a
436 syncrepl specification is included in a database definition,
437 {{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
438 and schedules its execution. If the {{refreshOnly}} operation is
439 specified, the syncrepl engine will be rescheduled at the interval
440 time after a synchronization operation is completed. If the
441 {{refreshAndPersist}} operation is specified, the engine will remain
442 active and process the persistent synchronization messages from the
445 The syncrepl engine utilizes both the present phase and the delete
446 phase of the refresh synchronization. It is possible to configure
447 a per-scope session log in the provider server which stores the
448 {{EX:entryUUID}}s of a finite number of entries deleted from a
449 replication content. Multiple replicas of single provider content
450 share the same per-scope session log. The syncrepl engine uses the
451 delete phase if the session log is present and the state of the
452 consumer server is recent enough that no session log entries are
453 truncated after the last synchronization of the client. The syncrepl
454 engine uses the present phase if no session log is configured for
455 the replication content or if the consumer replica is too outdated
456 to be covered by the session log. The current design of the session
457 log store is memory based, so the information contained in the
458 session log is not persistent over multiple provider invocations.
459 It is not currently supported to access the session log store by
460 using LDAP operations. It is also not currently supported to impose
461 access control to the session log.
463 As a further optimization, even in the case the synchronization
464 search is not associated with any session log, no entries will be
465 transmitted to the consumer server when there has been no update
466 in the replication context.
468 The syncrepl engine, which is a consumer-side replication engine,
469 can work with any backends. The LDAP Sync provider can be configured
470 as an overlay on any backend, but works best with the {{back-bdb}}
471 or {{back-hdb}} backend.
473 The LDAP Sync provider maintains a {{EX:contextCSN}} for each
474 database as the current synchronization state indicator of the
475 provider content. It is the largest {{EX:entryCSN}} in the provider
476 context such that no transactions for an entry having smaller
477 {{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}}
478 could not just be set to the largest issued {{EX:entryCSN}} because
479 {{EX:entryCSN}} is obtained before a transaction starts and
480 transactions are not committed in the issue order.
482 The provider stores the {{EX:contextCSN}} of a context in the
483 {{EX:contextCSN}} attribute of the context suffix entry. The attribute
484 is not written to the database after every update operation though;
485 instead it is maintained primarily in memory. At database start
486 time the provider reads the last saved {{EX:contextCSN}} into memory
487 and uses the in-memory copy exclusively thereafter. By default,
488 changes to the {{EX:contextCSN}} as a result of database updates
489 will not be written to the database until the server is cleanly
490 shut down. A checkpoint facility exists to cause the contextCSN to
491 be written out more frequently if desired.
493 Note that at startup time, if the provider is unable to read a
494 {{EX:contextCSN}} from the suffix entry, it will scan the entire
495 database to determine the value, and this scan may take quite a
496 long time on a large database. When a {{EX:contextCSN}} value is
497 read, the database will still be scanned for any {{EX:entryCSN}}
498 values greater than it, to make sure the {{EX:contextCSN}} value
499 truly reflects the greatest committed {{EX:entryCSN}} in the database.
500 On databases which support inequality indexing, setting an eq index
501 on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
502 checkpoints will greatly speed up this scanning step.
504 If no {{EX:contextCSN}} can be determined by reading and scanning
505 the database, a new value will be generated. Also, if scanning the
506 database yielded a greater {{EX:entryCSN}} than was previously
507 recorded in the suffix entry's {{EX:contextCSN}} attribute, a
508 checkpoint will be immediately written with the new value.
510 The consumer also stores its replica state, which is the provider's
511 {{EX:contextCSN}} received as a synchronization cookie, in the
512 {{EX:contextCSN}} attribute of the suffix entry. The replica state
513 maintained by a consumer server is used as the synchronization state
514 indicator when it performs subsequent incremental synchronization
515 with the provider server. It is also used as a provider-side
516 synchronization state indicator when it functions as a secondary
517 provider server in a cascading replication configuration. Since
518 the consumer and provider state information are maintained in the
519 same location within their respective databases, any consumer can
520 be promoted to a provider (and vice versa) without any special
523 Because a general search filter can be used in the syncrepl
524 specification, some entries in the context may be omitted from the
525 synchronization content. The syncrepl engine creates a glue entry
526 to fill in the holes in the replica context if any part of the
527 replica content is subordinate to the holes. The glue entries will
528 not be returned in the search result unless {{ManageDsaIT}} control
531 Also as a consequence of the search filter used in the syncrepl
532 specification, it is possible for a modification to remove an entry
533 from the replication scope even though the entry has not been deleted
534 on the provider. Logically the entry must be deleted on the consumer
535 but in {{refreshOnly}} mode the provider cannot detect and propagate
536 this change without the use of the session log.
538 For configuration, please see the {{SECT:Syncrepl}} section.
541 H3: Delta-syncrepl replication
543 * Disadvantages of Syncrepl replication:
545 OpenLDAP's syncrepl replication is an object-based replication mechanism.
546 When any attribute value in a replicated object is changed on the provider,
547 each consumer fetches and processes the complete changed object {{B:both changed and unchanged attribute values}} during replication. This works well, but has drawbacks in some situations.
549 For example, suppose you have a database consisting of 100,000 objects of 1 KB
550 each. Further, suppose you routinely run a batch job to change the value of
551 a single two-byte attribute value that appears in each of the 100,000 objects
552 on the master. Not counting LDAP and TCP/IP protocol overhead, each time you
553 run this job each consumer will transfer and process {{B:1 GB}} of data to process
554 {{B:200KB of changes!}}
556 99.98% of the data that is transmitted and processed in a case like this will
557 be redundant, since it represents values that did not change. This is a waste
558 of valuable transmission and processing bandwidth and can cause an unacceptable
559 replication backlog to develop. While this situation is extreme, it serves to
560 demonstrate a very real problem that is encountered in some LDAP deployments.
563 * Where Delta-syncrepl comes in:
565 Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address
566 situations like the one described above. Delta-syncrepl works by maintaining a
567 changelog of a selectable depth on the provider. The replication consumer on
568 each consumer checks the changelog for the changes it needs and, as long as
569 the changelog contains the needed changes, the delta-syncrepl consumer fetches
570 them from the changelog and applies them to its database. If, however, a replica
571 is too far out of sync (or completely empty), conventional syncrepl is used to
572 bring it up to date and replication then switches to the delta-syncrepl mode.
574 For configuration, please see the {{SECT:Delta-syncrepl}} section.
577 H2: Mixture of both Pull and Push based
579 H3: N-Way Multi-Master replication
581 Multi-Master replication is a replication technique using Syncrepl to replicate
582 data to multiple Master Directory servers.
584 * Advantages of Multi-Master replication:
586 - If any master fails, other masters will continue to accept updates
587 - Avoids a single point of failure
588 - Masters can be located in several physical sites i.e. distributed across the
590 - Good for Automatic failover/High Availability
592 * Disadvantages of Multi-Master replication:
594 - It has {{B:NOTHING}} to do with load balancing
595 - {{URL:http://www.openldap.org/faq/data/cache/1240.html}}
596 - If connectivity with a master is lost because of a network partition, then
597 "automatic failover" can just compound the problem
598 - Typically, a particular machine cannot distinguish between losing contact
599 with a peer because that peer crashed, or because the network link has failed
600 - If a network is partitioned and multiple clients start writing to each of the
601 "masters" then reconciliation will be a pain; it may be best to simply deny
602 writes to the clients that are partitioned from the single master
603 - Masters {{B:must}} propagate writes to {{B:all}} the other servers, which
604 means the network traffic and write load is constant and spreads across all
608 For configuration, please see the {{SECT:N-Way Multi-Master}} section below
610 H3: MirrorMode replication
612 MirrorMode is a hybrid configuration that provides all of the consistency
613 guarantees of single-master replication, while also providing the high
614 availability of multi-master. In MirrorMode two masters are set up to
615 replicate from each other (as a multi-master configuration) but an
616 external frontend is employed to direct all writes to only one of
617 the two servers. The second master will only be used for writes if
618 the first master crashes, at which point the frontend will switch to
619 directing all writes to the second master. When a crashed master is
620 repaired and restarted it will automatically catch up to any changes
621 on the running master and resync.
623 H4: Arguments for MirrorMode
625 * Provides a high-availability (HA) solution for directory writes (replicas handle reads)
626 * As long as one Master is operational, writes can safely be accepted
627 * Master nodes replicate from each other, so they are always up to date and
628 can be ready to take over (hot standby)
629 * Syncrepl also allows the master nodes to re-synchronize after any downtime
630 * Delta-Syncrepl can be used
633 H4: Arguments against MirrorMode
635 * MirrorMode is not what is termed as a Multi-Master solution. This is because
636 writes have to go to one of the mirror nodes at a time
637 * MirrorMode can be termed as Active-Active Hot-Standby, therefor an external
638 server (slapd in proxy mode) or device (hardware load balancer) to manage which
639 master is currently active
640 * While syncrepl can recover from a completely empty database, slapadd is much
642 * Does not provide faster or more scalable write performance (neither could
643 any Multi-Master solution)
644 * Backups are managed slightly differently
645 - If backing up the Berkeley database itself and periodically backing up the
646 transaction log files, then the same member of the mirror pair needs to be
647 used to collect logfiles until the next database backup is taken
648 - To ensure that both databases are consistent, each database might have to be
649 put in read-only mode while performing a slapcat.
650 - When using slapcat, the generated LDIF files can be rather large. This can
651 happen with a non-MirrorMode deployment also.
653 For configuration, please see the {{SECT:MirrorMode}} section below
656 H2: Configuring the different replication types
660 H4: Syncrepl configuration
662 Because syncrepl is a consumer-side replication engine, the syncrepl
663 specification is defined in {{slapd.conf}}(5) of the consumer
664 server, not in the provider server's configuration file. The initial
665 loading of the replica content can be performed either by starting
666 the syncrepl engine with no synchronization cookie or by populating
667 the consumer replica by adding an {{TERM:LDIF}} file dumped as a
668 backup at the provider.
670 When loading from a backup, it is not required to perform the initial
671 loading from the up-to-date backup of the provider content. The
672 syncrepl engine will automatically synchronize the initial consumer
673 replica to the current provider content. As a result, it is not
674 required to stop the provider server in order to avoid the replica
675 inconsistency caused by the updates to the provider content during
676 the content backup and loading process.
678 When replicating a large scale directory, especially in a bandwidth
679 constrained environment, it is advised to load the consumer replica
680 from a backup instead of performing a full initial load using
684 H4: Set up the provider slapd
686 The provider is implemented as an overlay, so the overlay itself
687 must first be configured in {{slapd.conf}}(5) before it can be
688 used. The provider has only two configuration directives, for setting
689 checkpoints on the {{EX:contextCSN}} and for configuring the session
690 log. Because the LDAP Sync search is subject to access control,
691 proper access control privileges should be set up for the replicated
694 The {{EX:contextCSN}} checkpoint is configured by the
696 > syncprov-checkpoint <ops> <minutes>
698 directive. Checkpoints are only tested after successful write
699 operations. If {{<ops>}} operations or more than {{<minutes>}}
700 time has passed since the last checkpoint, a new checkpoint is
703 The session log is configured by the
705 > syncprov-sessionlog <size>
707 directive, where {{<size>}} is the maximum number of session log
708 entries the session log can record. When a session log is configured,
709 it is automatically used for all LDAP Sync searches within the
712 Note that using the session log requires searching on the {{entryUUID}}
713 attribute. Setting an eq index on this attribute will greatly benefit
714 the performance of the session log on the provider.
716 A more complete example of the {{slapd.conf}}(5) content is thus:
719 > suffix dc=Example,dc=com
720 > rootdn dc=Example,dc=com
721 > directory /var/ldap/db
722 > index objectclass,entryCSN,entryUUID eq
725 > syncprov-checkpoint 100 10
726 > syncprov-sessionlog 100
729 H4: Set up the consumer slapd
731 The syncrepl replication is specified in the database section of
732 {{slapd.conf}}(5) for the replica context. The syncrepl engine
733 is backend independent and the directive can be defined with any
737 > suffix dc=Example,dc=com
738 > rootdn dc=Example,dc=com
739 > directory /var/ldap/db
740 > index objectclass,entryCSN,entryUUID eq
743 > provider=ldap://provider.example.com:389
745 > interval=01:00:00:00
746 > searchbase="dc=example,dc=com"
747 > filter="(objectClass=organizationalPerson)"
749 > attrs="cn,sn,ou,telephoneNumber,title,l"
752 > binddn="cn=syncuser,dc=example,dc=com"
755 In this example, the consumer will connect to the provider {{slapd}}(8)
756 at port 389 of {{FILE:ldap://provider.example.com}} to perform a
757 polling ({{refreshOnly}}) mode of synchronization once a day. It
758 will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
759 authentication with password "secret". Note that the access control
760 privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
761 appropriately in the provider to retrieve the desired replication
762 content. Also the search limits must be high enough on the provider
763 to allow the syncuser to retrieve a complete copy of the requested
764 content. The consumer uses the rootdn to write to its database so
765 it always has full permissions to write all content.
767 The synchronization search in the above example will search for the
768 entries whose objectClass is organizationalPerson in the entire
769 subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
770 are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
771 {{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
772 that the consumer {{slapd}}(8) will not enforce entry schema
773 checking when it process updates from the provider {{slapd}}(8).
775 For more detailed information on the syncrepl directive, see the
776 {{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
777 chapter of this admin guide.
780 H4: Start the provider and the consumer slapd
782 The provider {{slapd}}(8) is not required to be restarted.
783 {{contextCSN}} is automatically generated as needed: it might be
784 originally contained in the {{TERM:LDIF}} file, generated by
785 {{slapadd}} (8), generated upon changes in the context, or generated
786 when the first LDAP Sync search arrives at the provider. If an
787 LDIF file is being loaded which did not previously contain the
788 {{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
789 (8) to cause it to be generated. This will allow the server to
790 startup a little quicker the first time it runs.
792 When starting a consumer {{slapd}}(8), it is possible to provide
793 a synchronization cookie as the {{-c cookie}} command line option
794 in order to start the synchronization from a specific state. The
795 cookie is a comma separated list of name=value pairs. Currently
796 supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
797 {{<csn>}} represents the current synchronization state of the
798 consumer replica. {{<rid>}} identifies a consumer replica locally
799 within the consumer server. It is used to relate the cookie to the
800 syncrepl definition in {{slapd.conf}}(5) which has the matching
801 replica identifier. The {{<rid>}} must have no more than 3 decimal
802 digits. The command line cookie overrides the synchronization
803 cookie stored in the consumer replica database.
808 H4: Delta-syncrepl Master configuration
810 Setting up delta-syncrepl requires configuration changes on both the master and
813 > # Give the replica DN unlimited read access. This ACL may need to be
814 > # merged with other ACL statements.
817 > by dn.base="cn=replicator,dc=symas,dc=com" read
820 > # Set the module path location
821 > modulepath /opt/symas/lib/openldap
823 > # Load the hdb backend
824 > moduleload back_hdb.la
826 > # Load the accesslog overlay
827 > moduleload accesslog.la
829 > #Load the syncprov overlay
830 > moduleload syncprov.la
832 > # Accesslog database definitions
834 > suffix cn=accesslog
835 > directory /db/accesslog
836 > rootdn cn=accesslog
838 > index entryCSN,objectClass,reqEnd,reqResult,reqStart
841 > syncprov-nopresent TRUE
842 > syncprov-reloadhint TRUE
844 > # Let the replica DN have limitless searches
845 > limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
847 > # Primary database definitions
849 > suffix "dc=symas,dc=com"
850 > rootdn "cn=manager,dc=symas,dc=com"
852 > ## Whatever other configuration options are desired
854 > # syncprov specific indexing
858 > # syncrepl Provider for primary db
860 > syncprov-checkpoint 1000 60
862 > # accesslog overlay definitions for primary db
867 > # scan the accesslog DB every day, and purge entries older than 7 days
868 > logpurge 07+00:00 01+00:00
870 > # Let the replica DN have limitless searches
871 > limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
873 For more information, always consult the relevant man pages (slapo-accesslog and slapd.conf)
876 H4: Delta-syncrepl Replica configuration
878 > # Primary replica database configuration
880 > suffix "dc=symas,dc=com"
881 > rootdn "cn=manager,dc=symas,dc=com"
883 > ## Whatever other configuration bits for the replica, like indexing
886 > # syncrepl specific indices
889 > # syncrepl directives
891 > provider=ldap://ldapmaster.symas.com:389
893 > binddn="cn=replicator,dc=symas,dc=com"
895 > searchbase="dc=symas,dc=com"
896 > logbase="cn=accesslog"
897 > logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
899 > type=refreshAndPersist
903 > # Refer updates to the master
904 > updateref ldap://ldapmaster.symas.com
907 The above configuration assumes that you have a replicator identity defined
908 in your database that can be used to bind to the master with. In addition,
909 all of the databases (primary master, primary replica, and the accesslog
910 storage database) should also have properly tuned {{DB_CONFIG}} files that meet
914 H3: N-Way Multi-Master
916 For the following example we will be using 3 Master nodes. Keeping in line with
917 {{B:test050-syncrepl-multimaster}} of the OpenLDAP test suite, we will be configuring
918 {{slapd(8)}} via {{B:cn=config}}
920 This sets up the config database:
923 > objectClass: olcGlobal
927 > dn: olcDatabase={0}config,cn=config
928 > objectClass: olcDatabaseConfig
929 > olcDatabase: {0}config
932 second and third servers will have a different olcServerID obviously:
935 > objectClass: olcGlobal
939 > dn: olcDatabase={0}config,cn=config
940 > objectClass: olcDatabaseConfig
941 > olcDatabase: {0}config
944 This sets up syncrepl as a provider (since these are all masters):
946 > dn: cn=module,cn=config
947 > objectClass: olcModuleList
949 > olcModulePath: /usr/local/libexec/openldap
950 > olcModuleLoad: syncprov.la
952 Now we setup the first Master Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls):
956 > replace: olcServerID
957 > olcServerID: 1 $URI1
958 > olcServerID: 2 $URI2
959 > olcServerID: 3 $URI3
961 > dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
963 > objectClass: olcOverlayConfig
964 > objectClass: olcSyncProvConfig
965 > olcOverlay: syncprov
967 > dn: olcDatabase={0}config,cn=config
970 > olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
971 > credentials=secret searchbase="cn=config" type=refreshAndPersist
972 > retry="5 5 300 5" timeout=1
973 > olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
974 > credentials=secret searchbase="cn=config" type=refreshAndPersist
975 > retry="5 5 300 5" timeout=1
976 > olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
977 > credentials=secret searchbase="cn=config" type=refreshAndPersist
978 > retry="5 5 300 5" timeout=1
981 > olcMirrorMode: TRUE
983 Now start up the Master and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multimaster on the config database.
985 We still have to replicate the actual data, not just the config, so add to the master (all active and configured consumers/masters will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup:
987 > dn: olcDatabase={1}$BACKEND,cn=config
988 > objectClass: olcDatabaseConfig
989 > objectClass: olc${BACKEND}Config
990 > olcDatabase: {1}$BACKEND
992 > olcDbDirectory: ./db
993 > olcRootDN: $MANAGERDN
995 > olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple
996 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
997 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
998 > olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple
999 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
1000 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
1001 > olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple
1002 > credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
1003 > interval=00:00:00:10 retry="5 5 300 5" timeout=1
1004 > olcMirrorMode: TRUE
1006 > dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config
1008 > objectClass: olcOverlayConfig
1009 > objectClass: olcSyncProvConfig
1010 > olcOverlay: syncprov
1012 Note: You must have all your server set to the same time via {{http://www.ntp.org/}}
1016 MirrorMode configuration is actually very easy. If you have ever setup a normal
1017 slapd syncrepl provider, then the only change is the following two directives:
1022 Note: You need to make sure that the {{serverID}} of each mirror node pair is
1023 different and add it as a global configuration option.
1025 H4: Mirror Node Configuration
1027 This is the same as the {{SECT:Set up the provider slapd}} section.
1029 Note: Delta-syncrepl is not yet supported with MirrorMode.
1031 Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in
1032 {{refreshAndPersist}} mode:
1038 > # database section
1040 > # syncrepl directives
\r
1041 > syncrepl rid=001
\r
1042 > provider=ldap://ldap-ridr1.example.com
\r
1043 > bindmethod=simple
\r
1044 > binddn="cn=mirrormode,dc=example,dc=com"
\r
1045 > credentials=mirrormode
\r
1046 > searchbase="dc=example,dc=com"
\r
1047 > schemachecking=on
\r
1048 > type=refreshAndPersist
\r
1051 > syncrepl rid=002
\r
1052 > provider=ldap://ldap-rid2.example.com
\r
1053 > bindmethod=simple
\r
1054 > binddn="cn=mirrormode,dc=example,dc=com"
\r
1055 > credentials=mirrormode
\r
1056 > searchbase="dc=example,dc=com"
\r
1057 > schemachecking=on
\r
1058 > type=refreshAndPersist
\r
1067 > # database section
1069 > # syncrepl directives
\r
1070 > syncrepl rid=001
\r
1071 > provider=ldap://ldap-ridr1.example.com
\r
1072 > bindmethod=simple
\r
1073 > binddn="cn=mirrormode,dc=example,dc=com"
\r
1074 > credentials=mirrormode
\r
1075 > searchbase="dc=example,dc=com"
\r
1076 > schemachecking=on
\r
1077 > type=refreshAndPersist
\r
1080 > syncrepl rid=002
\r
1081 > provider=ldap://ldap-rid2.example.com
\r
1082 > bindmethod=simple
\r
1083 > binddn="cn=mirrormode,dc=example,dc=com"
\r
1084 > credentials=mirrormode
\r
1085 > searchbase="dc=example,dc=com"
\r
1086 > schemachecking=on
\r
1087 > type=refreshAndPersist
\r
1092 It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except
1093 that the {{serverID}} is unique.
1095 H5: Failover Configuration
1097 There are generally 2 choices for this; 1. Hardware proxies/load-balancing or
1098 dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider
1100 A typical enterprise example might be:
1102 !import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration"
1103 FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration
1105 H5: Normal Consumer Configuration
1107 This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It
1108 can either setup in normal {{SECT:syncrepl replication}} mode, or in
1109 {{SECT:delta-syncrepl replication}} mode.
1111 H4: MirrorMode Summary
1113 Hopefully you will now have a directory architecture that provides all of the
1114 consistency guarantees of single-master replication, whilst also providing the
1115 high availability of multi-master replication.