rigidly defined master/slave relationships, provider/consumer roles
are quite fluid: replication updates received in a consumer can be
further propagated by that consumer to other servers, so a consumer
-can also act simultaneously as a provider. The following sections will
+can also act simultaneously as a provider. Also, a consumer need not
+be an actual LDAP server; it may be just an LDAP client.
+
+The following sections will describe the replication technology and
discuss the various replication options that are available.
-H2: Pull Based
+H2: Replication Technology
H3: LDAP Sync Replication
The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
short, is a consumer-side replication engine that enables the
consumer {{TERM:LDAP}} server to maintain a shadow copy of a
-{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
+{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer
and executes as one of the {{slapd}}(8) threads. It creates and maintains a
consumer replica by connecting to the replication provider to perform
the initial DIT content load followed either by periodic content
polling or by timely updates upon content changes.
-Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
-short) protocol as the replica synchronization protocol. LDAP Sync provides
+Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for
+short) as the replica synchronization protocol. LDAP Sync provides
a stateful replication which supports both pull-based and push-based
synchronization and does not mandate the use of a history store.
+In pull-based replication the consumer periodically
+polls the provider for updates. In push-based replication the consumer
+listens for updates that are sent by the provider in realtime. Since the
+protocol does not require a history store, the provider does not need to
+maintain any log of updates it has received. (Note
+that the syncrepl engine is extensible and additional replication
+protocols may be supported in the future.)
Syncrepl keeps track of the status of the replication content by
maintaining and exchanging synchronization cookies. Because the
server can stop the replication also without the need for provider-side
changes and restart.
-Syncrepl supports both partial and sparse replications. The shadow
+Syncrepl supports partial, sparse, and fractional replications. The shadow
DIT fragment is defined by a general search criteria consisting of
base, scope, filter, and attribute list. The replica content is
also subject to the access privileges of the bind identity of the
operation. This section introduces the LDAP Content Sync protocol
only briefly. For more information, refer to {{REF:RFC4533}}.
-The LDAP Sync protocol supports both polling and listening for
-changes by defining two respective synchronization operations:
+The LDAP Sync protocol supports both polling and listening for changes
+by defining two respective synchronization operations:
{{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
-by the {{refreshOnly}} operation. The client copy is synchronized
-to the server copy at the time of polling. The server finishes the
+by the {{refreshOnly}} operation. The consumer
+polls the provider using an LDAP Search request with an LDAP Sync
+control attached. The consumer copy is synchronized
+to the provider copy at the time of polling using the information
+returned in the search. The provider finishes the
search operation by returning {{SearchResultDone}} at the end of
-the search operation as in the normal search. The listening is
-implemented by the {{refreshAndPersist}} operation. Instead of
+the search operation as in the normal search. Listening is
+implemented by the {{refreshAndPersist}} operation. As the name
+implies, it begins with a search, like refreshOnly. Instead of
finishing the search after returning all entries currently matching
the search criteria, the synchronization search remains persistent
-in the server. Subsequent updates to the synchronization content
-in the server cause additional entry updates to be sent to the
-client.
+in the provider. Subsequent updates to the synchronization content
+in the provider cause additional entry updates to be sent to the
+consumer.
The {{refreshOnly}} operation and the refresh stage of the
{{refreshAndPersist}} operation can be performed with a present
phase or a delete phase.
-In the present phase, the server sends the client the entries updated
-within the search scope since the last synchronization. The server
-sends all requested attributes, be it changed or not, of the updated
+In the present phase, the provider sends the consumer the entries updated
+within the search scope since the last synchronization. The provider
+sends all requested attributes, be they changed or not, of the updated
entries. For each unchanged entry which remains in the scope, the
-server sends a present message consisting only of the name of the
+provider sends a present message consisting only of the name of the
entry and the synchronization control representing state present.
The present message does not contain any attributes of the entry.
-After the client receives all update and present entries, it can
-reliably determine the new client copy by adding the entries added
-to the server, by replacing the entries modified at the server, and
-by deleting entries in the client copy which have not been updated
-nor specified as being present at the server.
+After the consumer receives all update and present entries, it can
+reliably determine the new consumer copy by adding the entries added
+to the provider, by replacing the entries modified at the provider, and
+by deleting entries in the consumer copy which have not been updated
+nor specified as being present at the provider.
The transmission of the updated entries in the delete phase is the
-same as in the present phase. The server sends all the requested
+same as in the present phase. The provider sends all the requested
attributes of the entries updated within the search scope since the
-last synchronization to the client. In the delete phase, however,
-the server sends a delete message for each entry deleted from the
+last synchronization to the consumer. In the delete phase, however,
+the provider sends a delete message for each entry deleted from the
search scope, instead of sending present messages. The delete
message consists only of the name of the entry and the synchronization
-control representing state delete. The new client copy can be
+control representing state delete. The new consumer copy can be
determined by adding, modifying, and removing entries according to
the synchronization control attached to the {{SearchResultEntry}}
message.
-In the case that the LDAP Sync server maintains a history store and
-can determine which entries are scoped out of the client copy since
-the last synchronization time, the server can use the delete phase.
-If the server does not maintain any history store, cannot determine
+In the case that the LDAP Sync provider maintains a history store and
+can determine which entries are scoped out of the consumer copy since
+the last synchronization time, the provider can use the delete phase.
+If the provider does not maintain any history store, cannot determine
the scoped-out entries from the history store, or the history store
-does not cover the outdated synchronization state of the client,
-the server should use the present phase. The use of the present
+does not cover the outdated synchronization state of the consumer,
+the provider should use the present phase. The use of the present
phase is much more efficient than a full content reload in terms
of the synchronization traffic. To reduce the synchronization
traffic further, the LDAP Sync protocol also provides several
and the transmission of multiple {{EX:entryUUIDs}} in a single
{{syncIdSet}} message.
-At the end of the {{refreshOnly}} synchronization, the server sends
-a synchronization cookie to the client as a state indicator of the
-client copy after the synchronization is completed. The client
+At the end of the {{refreshOnly}} synchronization, the provider sends
+a synchronization cookie to the consumer as a state indicator of the
+consumer copy after the synchronization is completed. The consumer
will present the received cookie when it requests the next incremental
-synchronization to the server.
+synchronization to the provider.
-When {{refreshAndPersist}} synchronization is used, the server sends
+When {{refreshAndPersist}} synchronization is used, the provider sends
a synchronization cookie at the end of the refresh stage by sending
-a Sync Info message with TRUE refreshDone. It also sends a
+a Sync Info message with refreshDone=TRUE. It also sends a
synchronization cookie by attaching it to {{SearchResultEntry}}
-generated in the persist stage of the synchronization search. During
-the persist stage, the server can also send a Sync Info message
-containing the synchronization cookie at any time the server wants
-to update the client-side state indicator. The server also updates
-a synchronization indicator of the client at the end of the persist
-stage.
+messages generated in the persist stage of the synchronization search. During
+the persist stage, the provider can also send a Sync Info message
+containing the synchronization cookie at any time the provider wants
+to update the consumer-side state indicator.
In the LDAP Sync protocol, entries are uniquely identified by the
{{EX:entryUUID}} attribute value. It can function as a reliable
{{SearchResultEntry}} or {{SearchResultReference}} as a part of the
synchronization control.
-
H4: Syncrepl Details
The syncrepl engine utilizes both the {{refreshOnly}} and the
The syncrepl engine utilizes both the present phase and the delete
phase of the refresh synchronization. It is possible to configure
-a per-scope session log in the provider server which stores the
+a session log in the provider which stores the
{{EX:entryUUID}}s of a finite number of entries deleted from a
-replication content. Multiple replicas of single provider content
-share the same per-scope session log. The syncrepl engine uses the
+database. Multiple replicas share the same session log. The syncrepl
+engine uses the
delete phase if the session log is present and the state of the
consumer server is recent enough that no session log entries are
truncated after the last synchronization of the client. The syncrepl
For configuration, please see the {{SECT:Syncrepl}} section.
+H2: Deployment Alternatives
+
+While the LDAP Sync specification only defines a narrow scope for replication,
+the OpenLDAP implementation is extremely flexible and supports a variety of
+operating modes to handle other scenarios not explicitly addressed in the spec.
+
+
H3: Delta-syncrepl replication
-* Disadvantages of Syncrepl replication:
+* Disadvantages of LDAP Sync replication:
-OpenLDAP's syncrepl replication is an object-based replication mechanism.
+LDAP Sync replication is an object-based replication mechanism.
When any attribute value in a replicated object is changed on the provider,
-each consumer fetches and processes the complete changed object {{B:both changed and unchanged attribute values}} during replication. This works well, but has drawbacks in some situations.
+each consumer fetches and processes the complete changed object
+{{B:both changed and unchanged attribute values}} during replication.
+One advantage of this approach is that when multiple changes occur to
+a single object, the precise sequence of those changes need not be preserved;
+only the final state of the entry is significant. But this approach
+may have drawbacks when the usage pattern involves single changes to
+multiple objects.
For example, suppose you have a database consisting of 100,000 objects of 1 KB
each. Further, suppose you routinely run a batch job to change the value of
a single two-byte attribute value that appears in each of the 100,000 objects
on the master. Not counting LDAP and TCP/IP protocol overhead, each time you
-run this job each consumer will transfer and process {{B:1 GB}} of data to process
-{{B:200KB of changes!}}
+run this job each consumer will transfer and process {{B:1 GB}} of data to
+process {{B:200KB of changes!}}
99.98% of the data that is transmitted and processed in a case like this will
be redundant, since it represents values that did not change. This is a waste
Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address
situations like the one described above. Delta-syncrepl works by maintaining a
-changelog of a selectable depth on the provider. The replication consumer on
-each consumer checks the changelog for the changes it needs and, as long as
-the changelog contains the needed changes, the delta-syncrepl consumer fetches
-them from the changelog and applies them to its database. If, however, a replica
+changelog of a selectable depth on the provider. The replication consumer
+checks the changelog for the changes it needs and, as long as
+the changelog contains the needed changes, the consumer fetches the changes
+from the changelog and applies them to its database. If, however, a replica
is too far out of sync (or completely empty), conventional syncrepl is used to
-bring it up to date and replication then switches to the delta-syncrepl mode.
+bring it up to date and replication then switches back to the delta-syncrepl
+mode.
For configuration, please see the {{SECT:Delta-syncrepl}} section.
-H2: Push Based
-
-
-H3: Replacing Slurpd
-
-{{Slurpd}} replication has been deprecated in favor of Syncrepl replication and
-has been completely removed from OpenLDAP 2.4.
-
-{{Why was it replaced?}}
-
-The {{slurpd}} daemon was the original replication mechanism inherited from
-UMich's LDAP and operates in push mode: the master pushes changes to the
-slaves. It has been replaced for many reasons, in brief:
-
- * It is not reliable
- * It is extremely sensitive to the ordering of records in the replog
- * It can easily go out of sync, at which point manual intervention is
- required to resync the slave database with the master directory
- * It isn't very tolerant of unavailable servers. If a slave goes down
- for a long time, the replog may grow to a size that's too large for
- slurpd to process
-
-{{What was it replaced with?}}
-
-Syncrepl
-
-{{Why is Syncrepl better?}}
-
- * Syncrepl is self-synchronizing; you can start with a database in any
- state from totally empty to fully synced and it will automatically do
- the right thing to achieve and maintain synchronization
- * Syncrepl can operate in either direction
- * Data updates can be minimal or maximal
-
-{{How do I implement a pushed based replication system using Syncrepl?}}
-
-The easiest way is to point an LDAP backend ({{SECT: Backends}} and {{slapd-ldap(8)}})
-to your slave directory and setup Syncrepl to point to your Master database.
-
-If you imagine Syncrepl pulling down changes from the Master server, and then
-pushing those changes out to your slave servers via {{slapd-ldap(8)}}. This is
-called Syncrepl Proxy Mode. You can also use Syncrepl Multi-proxy mode:
-
-!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode"
-FT[align="Center"] Figure X.Y: Replacing slurpd
-
-The following example is for a self-contained push-based replication solution:
-
-> #######################################################################
-> # Standard OpenLDAP Master/Provider
-> #######################################################################
->
-> include /usr/local/etc/openldap/schema/core.schema
-> include /usr/local/etc/openldap/schema/cosine.schema
-> include /usr/local/etc/openldap/schema/nis.schema
-> include /usr/local/etc/openldap/schema/inetorgperson.schema
->
-> include /usr/local/etc/openldap/slapd.acl
->
-> modulepath /usr/local/libexec/openldap
-> moduleload back_hdb.la
-> moduleload syncprov.la
-> moduleload back_monitor.la
-> moduleload back_ldap.la
->
-> pidfile /usr/local/var/slapd.pid
-> argsfile /usr/local/var/slapd.args
->
-> loglevel sync stats
->
-> database hdb
-> suffix "dc=suretecsystems,dc=com"
-> directory /usr/local/var/openldap-data
->
-> checkpoint 1024 5
-> cachesize 10000
-> idlcachesize 10000
->
-> index objectClass eq
-> # rest of indexes
-> index default sub
->
-> rootdn "cn=admin,dc=suretecsystems,dc=com"
-> rootpw testing
->
-> # syncprov specific indexing
-> index entryCSN eq
-> index entryUUID eq
->
-> # syncrepl Provider for primary db
-> overlay syncprov
-> syncprov-checkpoint 1000 60
->
-> # Let the replica DN have limitless searches
-> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
->
-> database monitor
->
-> database config
-> rootpw testing
->
-> ##############################################################################
-> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
-> ##############################################################################
->
-> database ldap
-> # ignore conflicts with other databases, as we need to push out to same suffix
-> hidden on
-> suffix "dc=suretecsystems,dc=com"
-> rootdn "cn=slapd-ldap"
-> uri ldap://localhost:9012/
->
-> lastmod on
->
-> # We don't need any access to this DSA
-> restrict all
->
-> acl-bind bindmethod=simple
-> binddn="cn=replicator,dc=suretecsystems,dc=com"
-> credentials=testing
->
-> syncrepl rid=001
-> provider=ldap://localhost:9011/
-> binddn="cn=replicator,dc=suretecsystems,dc=com"
-> bindmethod=simple
-> credentials=testing
-> searchbase="dc=suretecsystems,dc=com"
-> type=refreshAndPersist
-> retry="5 5 300 5"
->
-> overlay syncprov
-
-A replica configuration for this type of setup could be:
-
-> #######################################################################
-> # Standard OpenLDAP Slave without Syncrepl
-> #######################################################################
->
-> include /usr/local/etc/openldap/schema/core.schema
-> include /usr/local/etc/openldap/schema/cosine.schema
-> include /usr/local/etc/openldap/schema/nis.schema
-> include /usr/local/etc/openldap/schema/inetorgperson.schema
->
-> include /usr/local/etc/openldap/slapd.acl
->
-> modulepath /usr/local/libexec/openldap
-> moduleload back_hdb.la
-> moduleload syncprov.la
-> moduleload back_monitor.la
-> moduleload back_ldap.la
->
-> pidfile /usr/local/var/slapd.pid
-> argsfile /usr/local/var/slapd.args
->
-> loglevel sync stats
->
-> database hdb
-> suffix "dc=suretecsystems,dc=com"
-> directory /usr/local/var/openldap-slave/data
->
-> checkpoint 1024 5
-> cachesize 10000
-> idlcachesize 10000
->
-> index objectClass eq
-> # rest of indexes
-> index default sub
->
-> rootdn "cn=admin,dc=suretecsystems,dc=com"
-> rootpw testing
->
-> # Let the replica DN have limitless searches
-> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
->
-> updatedn "cn=replicator,dc=suretecsystems,dc=com"
->
-> # Refer updates to the master
-> updateref ldap://localhost:9011
->
-> database monitor
->
-> database config
-> rootpw testing
-
-You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be:
-
-> # Give the replica DN unlimited read access. This ACL may need to be
-> # merged with other ACL statements.
->
-> access to *
-> by dn.base="cn=replicator,dc=suretecsystems,dc=com" write
-> by * break
->
-> access to dn.base=""
-> by * read
->
-> access to dn.base="cn=Subschema"
-> by * read
->
-> access to dn.subtree="cn=Monitor"
-> by dn.exact="uid=admin,dc=suretecsystems,dc=com" write
-> by users read
-> by * none
->
-> access to *
-> by self write
-> by * read
-
-In order to support more replicas, just add more {{database ldap}} sections and
-increment the {{syncrepl rid}} number accordingly.
-
-Note: You must populate the Master and Slave directories with the same data,
-unlike when using normal Syncrepl
-
-If you do not have access to modify the master directory configuration you can
-configure a standalone ldap proxy, which might look like:
-
-!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode"
-FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version
-
-The following configuration is an example of a standalone LDAP Proxy:
-
-> include /usr/local/etc/openldap/schema/core.schema
-> include /usr/local/etc/openldap/schema/cosine.schema
-> include /usr/local/etc/openldap/schema/nis.schema
-> include /usr/local/etc/openldap/schema/inetorgperson.schema
->
-> include /usr/local/etc/openldap/slapd.acl
->
-> modulepath /usr/local/libexec/openldap
-> moduleload syncprov.la
-> moduleload back_ldap.la
->
-> ##############################################################################
-> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
-> ##############################################################################
->
-> database ldap
-> # ignore conflicts with other databases, as we need to push out to same suffix
-> hidden on
-> suffix "dc=suretecsystems,dc=com"
-> rootdn "cn=slapd-ldap"
-> uri ldap://localhost:9012/
->
-> lastmod on
->
-> # We don't need any access to this DSA
-> restrict all
->
-> acl-bind bindmethod=simple
-> binddn="cn=replicator,dc=suretecsystems,dc=com"
-> credentials=testing
->
-> syncrepl rid=001
-> provider=ldap://localhost:9011/
-> binddn="cn=replicator,dc=suretecsystems,dc=com"
-> bindmethod=simple
-> credentials=testing
-> searchbase="dc=suretecsystems,dc=com"
-> type=refreshAndPersist
-> retry="5 5 300 5"
->
-> overlay syncprov
-
-As you can see, you can let your imagination go wild using Syncrepl and
-{{slapd-ldap(8)}} tailoring your replication to fit your specific network
-topology.
+H3: N-Way Multi-Master replication
+Multi-Master replication is a replication technique using Syncrepl to replicate
+data to multiple provider ("Master") Directory servers.
-H2: Mixture of both Pull and Push based
+H4: Valid Arguments for Multi-Master replication
-H3: N-Way Multi-Master replication
+* If any provider fails, other providers will continue to accept updates
+* Avoids a single point of failure
+* Providers can be located in several physical sites i.e. distributed across
+the network/globe.
+* Good for Automatic failover/High Availability
-Multi-Master replication is a replication technique using Syncrepl to replicate
-data to multiple Master Directory servers.
+H4: Invalid Arguments for Multi-Master replication
-* Advantages of Multi-Master replication:
+(These are often claimed to be advantages of Multi-Master replication but
+those claims are false):
-- If any master fails, other masters will continue to accept updates
-- Avoids a single point of failure
-- Masters can be located in several physical sites i.e. distributed across the
-network/globe.
-- Good for Automatic failover/High Availability
+* It has {{B:NOTHING}} to do with load balancing
+* Providers {{B:must}} propagate writes to {{B:all}} the other servers, which
+means the network traffic and write load spreads across all
+of the servers the same as for single-master.
+* Server utilization and performance are at best identical for
+Multi-Master and Single-Master replication; at worst Single-Master is
+superior because indexing can be tuned differently to optimize for the
+different usage patterns between the provider and the consumers.
-* Disadvantages of Multi-Master replication:
+H4: Arguments against Multi-Master replication
-- It has {{B:NOTHING}} to do with load balancing
-- {{URL:http://www.openldap.org/faq/data/cache/1240.html}}
-- If connectivity with a master is lost because of a network partition, then
+* Breaks the data consistency guarantees of the directory model
+* {{URL:http://www.openldap.org/faq/data/cache/1240.html}}
+* If connectivity with a provider is lost because of a network partition, then
"automatic failover" can just compound the problem
-- Typically, a particular machine cannot distinguish between losing contact
+* Typically, a particular machine cannot distinguish between losing contact
with a peer because that peer crashed, or because the network link has failed
-- If a network is partitioned and multiple clients start writing to each of the
+* If a network is partitioned and multiple clients start writing to each of the
"masters" then reconciliation will be a pain; it may be best to simply deny
-writes to the clients that are partitioned from the single master
-- Masters {{B:must}} propagate writes to {{B:all}} the other servers, which
-means the network traffic and write load is constant and spreads across all
-of the servers
+writes to the clients that are partitioned from the single provider
For configuration, please see the {{SECT:N-Way Multi-Master}} section below
MirrorMode is a hybrid configuration that provides all of the consistency
guarantees of single-master replication, while also providing the high
-availability of multi-master. In MirrorMode two masters are set up to
+availability of multi-master. In MirrorMode two providers are set up to
replicate from each other (as a multi-master configuration), but an
external frontend is employed to direct all writes to only one of
-the two servers. The second master will only be used for writes if
-the first master crashes, at which point the frontend will switch to
-directing all writes to the second master. When a crashed master is
+the two servers. The second provider will only be used for writes if
+the first provider crashes, at which point the frontend will switch to
+directing all writes to the second provider. When a crashed provider is
repaired and restarted it will automatically catch up to any changes
-on the running master and resync.
+on the running provider and resync.
H4: Arguments for MirrorMode
* Provides a high-availability (HA) solution for directory writes (replicas handle reads)
-* As long as one Master is operational, writes can safely be accepted
-* Master nodes replicate from each other, so they are always up to date and
+* As long as one provider is operational, writes can safely be accepted
+* Provider nodes replicate from each other, so they are always up to date and
can be ready to take over (hot standby)
-* Syncrepl also allows the master nodes to re-synchronize after any downtime
+* Syncrepl also allows the provider nodes to re-synchronize after any downtime
* Delta-Syncrepl can be used
H4: Arguments against MirrorMode
* MirrorMode is not what is termed as a Multi-Master solution. This is because
-writes have to go to one of the mirror nodes at a time
-* MirrorMode can be termed as Active-Active Hot-Standby, therefor an external
-server (slapd in proxy mode) or device (hardware load balancer) to manage which
-master is currently active
-* While syncrepl can recover from a completely empty database, slapadd is much
-faster
-* Does not provide faster or more scalable write performance (neither could
- any Multi-Master solution)
+writes have to go to just one of the mirror nodes at a time
+* MirrorMode can be termed as Active-Active Hot-Standby, therefore an external
+server (slapd in proxy mode) or device (hardware load balancer)
+is needed to manage which provider is currently active
* Backups are managed slightly differently
- If backing up the Berkeley database itself and periodically backing up the
transaction log files, then the same member of the mirror pair needs to be
used to collect logfiles until the next database backup is taken
- To ensure that both databases are consistent, each database might have to be
put in read-only mode while performing a slapcat.
-- When using slapcat, the generated LDIF files can be rather large. This can
-happen with a non-MirrorMode deployment also.
For configuration, please see the {{SECT:MirrorMode}} section below
+H3: Syncrepl Proxy Mode
+
+While the LDAP Sync protocol supports both pull- and push-based replication,
+the push mode (refreshAndPersist) must still be initiated from the consumer
+before the provider can begin pushing changes. In some network configurations,
+particularly where firewalls restrict the direction in which connections
+can be made, a provider-initiated push mode may be needed.
+
+This mode can be configured with the aid of the LDAP Backend
+({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the
+syncrepl engine on the actual consumer, a slapd-ldap proxy is set up
+near (or colocated with) the provider that points to the consumer,
+and the syncrepl engine runs on the proxy.
+
+For configuration, please see the {{SECT:Syncrepl Proxy}} section.
+
+H4: Replacing Slurpd
+
+The old {{slurpd}} mechanism only operated in provider-initiated
+push mode. Slurpd replication was deprecated in favor of Syncrepl
+replication and has been completely removed from OpenLDAP 2.4.
+
+The slurpd daemon was the original replication mechanism inherited from
+UMich's LDAP and operated in push mode: the master pushed changes to the
+slaves. It was replaced for many reasons, in brief:
+
+ * It was not reliable
+ ** It was extremely sensitive to the ordering of records in the replog
+ ** It could easily go out of sync, at which point manual intervention was
+ required to resync the slave database with the master directory
+ ** It wasn't very tolerant of unavailable servers. If a slave went down
+ for a long time, the replog could grow to a size that was too large for
+ slurpd to process
+ * It only worked in push mode
+ * It required stopping and restarting the master to add new slaves
+ * It only supported single master replication
+
+Syncrepl has none of those weaknesses:
+
+ * Syncrepl is self-synchronizing; you can start with a consumer database
+ in any state from totally empty to fully synced and it will automatically
+ do the right thing to achieve and maintain synchronization
+ ** It is completely insensitive to the order in which changes occur
+ ** It guarantees convergence between the consumer and the provider
+ content without manual intervention
+ ** It can resynchronize regardless of how long a consumer stays out
+ of contact with the provider
+ * Syncrepl can operate in either direction
+ * Consumers can be added at any time without touching anything on the
+ provider
+ * Multi-master replication is supported
+
+
H2: Configuring the different replication types
H3: Syncrepl
server, not in the provider server's configuration file. The initial
loading of the replica content can be performed either by starting
the syncrepl engine with no synchronization cookie or by populating
-the consumer replica by adding an {{TERM:LDIF}} file dumped as a
+the consumer replica by loading an {{TERM:LDIF}} file dumped as a
backup at the provider.
When loading from a backup, it is not required to perform the initial
are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
{{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
that the consumer {{slapd}}(8) will not enforce entry schema
-checking when it process updates from the provider {{slapd}}(8).
+checking when it processes updates from the provider {{slapd}}(8).
For more detailed information on the syncrepl directive, see the
{{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
H3: Delta-syncrepl
-H4: Delta-syncrepl Master configuration
+H4: Delta-syncrepl Provider configuration
Setting up delta-syncrepl requires configuration changes on both the master and
replica servers:
> # Let the replica DN have limitless searches
> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
-For more information, always consult the relevant man pages (slapo-accesslog and slapd.conf)
+For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5))
-H4: Delta-syncrepl Replica configuration
+H4: Delta-syncrepl Consumer configuration
-> # Primary replica database configuration
+> # Replica database configuration
> database hdb
> suffix "dc=symas,dc=com"
> rootdn "cn=manager,dc=symas,dc=com"
The above configuration assumes that you have a replicator identity defined
-in your database that can be used to bind to the master with. In addition,
-all of the databases (primary master, primary replica, and the accesslog
+in your database that can be used to bind to the provider. In addition,
+all of the databases (primary, replica, and the accesslog
storage database) should also have properly tuned {{DB_CONFIG}} files that meet
your needs.
> olcDbDirectory: ./db
> olcRootDN: $MANAGERDN
> olcRootPW: $PASSWD
+> olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
> olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple
> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
> interval=00:00:00:10 retry="5 5 300 5" timeout=1
> objectClass: olcSyncProvConfig
> olcOverlay: syncprov
-Note: You must have all your server set to the same time via {{http://www.ntp.org/}}
+Note: You must have all your servers set to the same time via {{http://www.ntp.org/}}
H3: MirrorMode
> mirrormode on
> serverID 1
-Note: You need to make sure that the {{serverID}} of each mirror node pair is
+Note: You need to make sure that the {{serverID}} of each mirror node is
different and add it as a global configuration option.
H4: Mirror Node Configuration
high availability of multi-master replication.
+H3: Syncrepl Proxy
+
+!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode"
+FT[align="Center"] Figure X.Y: Replacing slurpd
+
+The following example is for a self-contained push-based replication solution:
+
+> #######################################################################
+> # Standard OpenLDAP Master/Provider
+> #######################################################################
+>
+> include /usr/local/etc/openldap/schema/core.schema
+> include /usr/local/etc/openldap/schema/cosine.schema
+> include /usr/local/etc/openldap/schema/nis.schema
+> include /usr/local/etc/openldap/schema/inetorgperson.schema
+>
+> include /usr/local/etc/openldap/slapd.acl
+>
+> modulepath /usr/local/libexec/openldap
+> moduleload back_hdb.la
+> moduleload syncprov.la
+> moduleload back_monitor.la
+> moduleload back_ldap.la
+>
+> pidfile /usr/local/var/slapd.pid
+> argsfile /usr/local/var/slapd.args
+>
+> loglevel sync stats
+>
+> database hdb
+> suffix "dc=suretecsystems,dc=com"
+> directory /usr/local/var/openldap-data
+>
+> checkpoint 1024 5
+> cachesize 10000
+> idlcachesize 10000
+>
+> index objectClass eq
+> # rest of indexes
+> index default sub
+>
+> rootdn "cn=admin,dc=suretecsystems,dc=com"
+> rootpw testing
+>
+> # syncprov specific indexing
+> index entryCSN eq
+> index entryUUID eq
+>
+> # syncrepl Provider for primary db
+> overlay syncprov
+> syncprov-checkpoint 1000 60
+>
+> # Let the replica DN have limitless searches
+> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
+>
+> database monitor
+>
+> database config
+> rootpw testing
+>
+> ##############################################################################
+> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
+> ##############################################################################
+>
+> database ldap
+> # ignore conflicts with other databases, as we need to push out to same suffix
+> hidden on
+> suffix "dc=suretecsystems,dc=com"
+> rootdn "cn=slapd-ldap"
+> uri ldap://localhost:9012/
+>
+> lastmod on
+>
+> # We don't need any access to this DSA
+> restrict all
+>
+> acl-bind bindmethod=simple
+> binddn="cn=replicator,dc=suretecsystems,dc=com"
+> credentials=testing
+>
+> syncrepl rid=001
+> provider=ldap://localhost:9011/
+> binddn="cn=replicator,dc=suretecsystems,dc=com"
+> bindmethod=simple
+> credentials=testing
+> searchbase="dc=suretecsystems,dc=com"
+> type=refreshAndPersist
+> retry="5 5 300 5"
+>
+> overlay syncprov
+
+A replica configuration for this type of setup could be:
+
+> #######################################################################
+> # Standard OpenLDAP Slave without Syncrepl
+> #######################################################################
+>
+> include /usr/local/etc/openldap/schema/core.schema
+> include /usr/local/etc/openldap/schema/cosine.schema
+> include /usr/local/etc/openldap/schema/nis.schema
+> include /usr/local/etc/openldap/schema/inetorgperson.schema
+>
+> include /usr/local/etc/openldap/slapd.acl
+>
+> modulepath /usr/local/libexec/openldap
+> moduleload back_hdb.la
+> moduleload syncprov.la
+> moduleload back_monitor.la
+> moduleload back_ldap.la
+>
+> pidfile /usr/local/var/slapd.pid
+> argsfile /usr/local/var/slapd.args
+>
+> loglevel sync stats
+>
+> database hdb
+> suffix "dc=suretecsystems,dc=com"
+> directory /usr/local/var/openldap-slave/data
+>
+> checkpoint 1024 5
+> cachesize 10000
+> idlcachesize 10000
+>
+> index objectClass eq
+> # rest of indexes
+> index default sub
+>
+> rootdn "cn=admin,dc=suretecsystems,dc=com"
+> rootpw testing
+>
+> # Let the replica DN have limitless searches
+> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
+>
+> updatedn "cn=replicator,dc=suretecsystems,dc=com"
+>
+> # Refer updates to the master
+> updateref ldap://localhost:9011
+>
+> database monitor
+>
+> database config
+> rootpw testing
+
+You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be:
+
+> # Give the replica DN unlimited read access. This ACL may need to be
+> # merged with other ACL statements.
+>
+> access to *
+> by dn.base="cn=replicator,dc=suretecsystems,dc=com" write
+> by * break
+>
+> access to dn.base=""
+> by * read
+>
+> access to dn.base="cn=Subschema"
+> by * read
+>
+> access to dn.subtree="cn=Monitor"
+> by dn.exact="uid=admin,dc=suretecsystems,dc=com" write
+> by users read
+> by * none
+>
+> access to *
+> by self write
+> by * read
+
+In order to support more replicas, just add more {{database ldap}} sections and
+increment the {{syncrepl rid}} number accordingly.
+
+Note: You must populate the Master and Slave directories with the same data,
+unlike when using normal Syncrepl
+
+If you do not have access to modify the master directory configuration you can
+configure a standalone ldap proxy, which might look like:
+
+!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode"
+FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version
+
+The following configuration is an example of a standalone LDAP Proxy:
+
+> include /usr/local/etc/openldap/schema/core.schema
+> include /usr/local/etc/openldap/schema/cosine.schema
+> include /usr/local/etc/openldap/schema/nis.schema
+> include /usr/local/etc/openldap/schema/inetorgperson.schema
+>
+> include /usr/local/etc/openldap/slapd.acl
+>
+> modulepath /usr/local/libexec/openldap
+> moduleload syncprov.la
+> moduleload back_ldap.la
+>
+> ##############################################################################
+> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
+> ##############################################################################
+>
+> database ldap
+> # ignore conflicts with other databases, as we need to push out to same suffix
+> hidden on
+> suffix "dc=suretecsystems,dc=com"
+> rootdn "cn=slapd-ldap"
+> uri ldap://localhost:9012/
+>
+> lastmod on
+>
+> # We don't need any access to this DSA
+> restrict all
+>
+> acl-bind bindmethod=simple
+> binddn="cn=replicator,dc=suretecsystems,dc=com"
+> credentials=testing
+>
+> syncrepl rid=001
+> provider=ldap://localhost:9011/
+> binddn="cn=replicator,dc=suretecsystems,dc=com"
+> bindmethod=simple
+> credentials=testing
+> searchbase="dc=suretecsystems,dc=com"
+> type=refreshAndPersist
+> retry="5 5 300 5"
+>
+> overlay syncprov
+
+As you can see, you can let your imagination go wild using Syncrepl and
+{{slapd-ldap(8)}} tailoring your replication to fit your specific network
+topology.
projects / openldap / commitdiff