2 # Copyright 1999-2000, The OpenLDAP Foundation, All Rights Reserved.
3 # COPYING RESTRICTIONS APPLY, see COPYRIGHT.
5 H1: Database Creation and Maintenance Tools
7 This section tells you how to create a slapd database from
8 scratch, and how to do trouble shooting if you run into
9 problems. There are two ways to create a database. First,
10 you can create the database on-line using LDAP. With this
11 method, you simply start up slapd and add entries using the
12 LDAP client of your choice. This method is fine for relatively
13 small databases (a few hundred or thousand entries,
14 depending on your requirements).
16 The second method of database creation is to do it off-line,
17 using the index generation tools. This method is best if you
18 have many thousands of entries to create, which would take
19 an unacceptably long time using the LDAP method, or if you
20 want to ensure the database is not accessed while it is
24 H2: Creating a database over LDAP
26 With this method, you use the LDAP client of your choice
27 (e.g., the ldapadd(1) tool) to add entries, just like you would
28 once the database is created. You should be sure to set the
29 following configuration options before starting slapd:
33 As described in the preceding section, this option says what
34 entries are to be held by this database. You should set this
35 to the DN of the root of the subtree you are trying to create.
38 E: suffix "dc=OpenLDAP, dc=org"
40 You should be sure to specify a directory where the index
41 files should be created:
43 E: directory <directory>
47 E: directory /usr/local/openldap/slapd
49 You need to make it so you can connect to slapd as
50 somebody with permission to add entries. This is done
51 through the following two options in the database definition:
56 These options specify a DN and password that can be used
57 to authenticate as the "superuser" entry of the database (i.e.,
58 the entry allowed to do anything). The DN and password
59 specified here will always work, regardless of whether the
60 entry named actually exists or has the password given. This
61 solves the chicken-and-egg problem of how to authenticate
62 and add entries before any entries yet exist.
64 Finally, you should make sure that the database definition
65 contains the index definitions you want:
67 E: index {<attrlist> | default} [pres,eq,approx,sub,none]
69 For example, to index the cn, sn, uid and objectclass
70 attributes the following index configuration lines could be
74 E: index objectclass pres,eq
77 See Section 4 on the configuration file for more details on
78 this option. Once you have configured things to your liking,
79 start up slapd, connect with your LDAP client, and start
80 adding entries. For example, to add a the organizational entry
81 followed by a Postmaster entry using the {{I:ldapadd}} tool, you
82 could create a file called {{EX:/tmp/newentry}} with the contents:
85 E: dc=OpenLDAP, dc=org
86 E: objectClass=dcObject
87 E: objectClass=organization
91 E: o=OpenLDAP Foundation
92 E: description=The OpenLDAP Foundation
93 E: description=The OpenLDAP Project
95 E: cn=Postmaster, dc=OpenLDAP, dc=org
96 E: objectClass=organizationalRole
98 E: description=OpenLDAP Postmaster <Postmaster@OpenLDAP.org>
100 and then use a command like this to actually create the
103 E: ldapadd -f /tmp/newentry -D "cn=Manager, dc=OpenLDAP, dc=org" -w secret
105 The above command assumes that you have set {{EX: rootdn}} to
106 "cn=Manager, dc=OpenLDAP, dc=org" and {{EX: rootpw}}
110 H2: Creating a database off-line
112 The second method of database creation is to do it off-line,
113 using the index generation tools described below. This
114 method is best if you have many thousands of entries to
115 create, which would take an unacceptably long time using
116 the LDAP method described above. These tools read the
117 slapd configuration file and an input file containing a text
118 representation of the entries to add. They produce the LDBM
119 index files directly. There are several important configuration
120 options you will want to be sure and set in the config file
121 database definition first:
125 As described in the preceding section, this option says what
126 entries are to be held by this database. You should set this
127 to the DN of the root of the subtree you are trying to create.
130 E: suffix "dc=OpenLDAP, dc=org"
132 You should be sure to specify a directory where the index
133 files should be created:
135 E: directory <directory>
139 E: directory /usr/local/var/openldap
141 Next, you probably want to increase the size of the in-core
142 cache used by each open index file. For best performance
143 during index creation, the entire index should fit in memory. If
144 your data is too big for this, or your memory too small, you
145 can still make it pretty big and let the paging system do the
146 work. This size is set with the following option:
148 E: dbcachesize <integer>
152 E: dbcachesize 50000000
154 This would create a cache 50 MB big, which is pretty big (at
155 U-M, our database has about 125K entries, and our biggest
156 index file is about 45 MB). Experiment with this number a bit,
157 and the degree of parallelism (explained below), to see what
158 works best for your system. Remember to turn this number
159 back down once your index files are created and before you
162 Finally, you need to specify which indexes you want to build.
163 This is done by one or more index options.
165 E: index {<attrlist> | default} [pres,eq,approx,sub,none]
169 E: index cn,sn,uid pres,eq,approx
170 E: index default none
172 This would create presence, equality and approximate
173 indexes for the cn, sn, and uid attributes, and no indexes for
174 any other attributes. See the configuration file section for
175 more information on this option.
177 H3: The {{EX: ldif2ldbm}} program
179 Once you've configured things to your liking, you create the
180 indexes by running the ldif2ldbm program:
182 E: ldif2ldbm -i <inputfile> -f <slapdconfigfile>
183 E: [-d <debuglevel>] [-j <integer>]
184 E: [-n <databasenumber>] [-e <etcdir>]
186 The arguments have the following meanings:
190 Specifies the LDIF input file containing the entries to add in
191 text form (described below in Section 8.3).
193 E: -f <slapdconfigfile>
195 Specifies the slapd configuration file that tells where to
196 create the indexes, what indexes to create, etc.
200 Turn on debugging, as specified by {{EX: <debuglevel>}}. The
201 debug levels are the same as for slapd (see Section 6.1).
205 An optional argument that specifies that at most {{EX: <integer>}}
206 processes should be started in parallel when building the
207 indexes. The default is 1. If set to a value greater than one,
208 {{I: ldif2ldbm}} will create at most that many subprocesses at a
209 time when building the indexes. A separate subprocess is
210 created to build each attribute index. Running these
211 processes in parallel can speed things up greatly, but
212 beware of creating too many processes, all competing for
213 memory and disk resources.
215 E: -n <databasenumber>
217 An optional argument that specifies the configuration file
218 database for which to build indices. The first database listed
219 is "1", the second "2", etc. By default, the first ldbm database
220 in the configuration file is used.
224 An optional argument that specifies the directory where
225 {{EX: ldif2ldbm}} can find the other database conversion tools it
226 needs to execute ({{EX: ldif2index}} and friends). The default is the
227 installation {{EX: ETCDIR}}.
229 The next sections describe the programs invoked by
230 {{I: ldif2ldbm}} when it is building indexes. Normally, these
231 programs are invoked for you, but occasionally you may
232 want to invoke them yourself.
236 H3: The {{EX: ldif2index}} program
238 Sometimes it may be necessary to create a new attribute
239 index file without disturbing the rest of the database. This is
240 possible using the {{EX: ldif2index}} program. {{EX: ldif2index}} is invoked
243 E: ldif2index -i <inputfile> -f <slapdconfigfile>
244 E: [-d <debuglevel>] [-n <databasenumber>] <attr>
246 Where the -i, -f, -d, and -n options are the same as for the
247 {{I: ldif2ldbm}} program. {{EX: <attr>}} is the attribute to build an index for.
248 Which indexes are built (e.g., equality, substring, etc.) is
249 controlled by the corresponding index line in the slapd
252 You can use the ldbmcat program to create a suitable LDIF
253 input file from an existing LDBM database.
257 H3: The {{EX: ldif2id2entry}} program
259 The {{EX: ldif2id2entry}} program is normally invoked from {{EX: ldif2ldbm}}.
260 It is used to convert an LDIF text file into an {{EX: id2entry}} index.
261 It is unlikely that you would need to invoke it yourself, but if
262 you do it works like this
264 E: ldif2id2entry -i <inputfile> -f <slapdconfigfile>
265 E: [-d <debuglevel>] [-n <databasenumber>]
267 The arguments are the same as for the {{EX: ldif2ldbm}} program.
271 H3: The {{EX: ldif2id2children}} program
273 The {{EX: ldif2id2children}} program is normally invoked from
274 {{EX: ldif2ldbm}}. It is used to convert an LDIF text file into
275 {{EX: id2children}} and {{EX: dn2id}} indexes. Occasionally, it may be
276 necessary to run this program yourself, for example if one of
277 these indexes has become corrupted. {{EX: ldif2id2children}} is
280 E: ldif2id2children -i <inputfile> -f <slapdconfigfile>
281 E: [-d <debuglevel>] [-n <databasenumber>]
283 The arguments are the same as for the {{EX: ldif2ldbm}} program.
284 You can use the ldbmcat program to create a suitable LDIF
285 input file from an existing LDBM database.
289 H3: The {{EX: ldbmcat}} program
291 The {{EX: ldbmcat}} program is used to convert an {{EX: id2entry}} index
292 back into its LDIF text format. This can be useful when you
293 want to make a human-readable backup of your database,
294 or as an intermediate step in creating a new index using the
295 {{EX: ldif2index}} program. The program is invoked like this:
297 E: ldbmcat [-n] <filename>
299 where {{EX: <filename>}} is the name of the {{EX: id2entry}} index file. The
300 corresponding LDIF output is written to standard output.
302 The -n option can be used to prevent the printing of entry
303 IDs in the LDIF format. If you are creating an LDIF format for
304 use as input to {{EX: ldif2index}} or anything by {{EX: ldif2ldbm}}, you
305 should not use the -n option (because the entry IDs must
306 match those already in the id2entry file). If you are just
307 making a backup of your data, you can use the -n option to
312 H3: The {{EX: ldif}} program
314 The ldif program is used to convert arbitrary data values to
315 LDIF format. This can be useful when writing a program or
316 script to create the LDIF file you will feed into the ldif2ldbm
317 program, or when writing a SHELL backend. ldif takes an
318 attribute name as an argument, and reads the attribute
319 value(s) from standard input. It produces the LDIF formatted
320 attribute line(s) on standard output. The usage is:
322 E: ldif [-b] <attrname>
324 where {{EX: <attrname>}} is the name of the attribute. Without the
325 -b option, ldif considers each line of standard input to be a
326 separate value of the attribute.
328 The -b option can be used to force ldif to interpret its input
329 as a single raw binary value. This option is useful when
330 converting binary data such as a {{EX: jpegPhoto}} or {{EX: audio}}
334 H2: The LDIF text entry format
336 The LDAP Data Interchange Format (LDIF) is used to
337 represent LDAP entries in a simple text format. The basic
341 E: dn: <distinguished name>
342 E: <attrtype>: <attrvalue>
343 E: <attrtype>: <attrvalue>
347 where {{EX: <id>}} is the optional entry ID (a positive decimal
348 number). Normally, you would not supply the {{EX: <id>}}, allowing
349 the database creation tools to do that for you. The ldbmcat
350 program, however, produces an LDIF format that includes
351 {{EX: <id>}} so that new indexes created will be consistent.
353 A line may be continued by starting the next line with a
354 single space or tab character. e.g.,
356 E: dn: cn=Barbara J Jensen, dc=OpenLDAP, dc=org
358 Multiple attribute values are specified on separate lines. e.g.,
360 E: cn: Barbara J Jensen
363 If an {{EX: <attrvalue>}} contains a non-printing character, or
364 begins with a space or a colon `:', the {{EX: <attrtype>}} is followed
365 by a double colon and the value is encoded in base 64
366 notation. e.g., the value " begins with a space" would be
369 E: cn:: IGJlZ2lucyB3aXRoIGEgc3BhY2U=
371 Multiple entries within the same LDIF file are separated by
372 blank lines. Here's an example of an LDIF file containing
375 E: dn: cn=Barbara J Jensen, dc=OpenLDAP, dc=org
376 E: cn: Barbara J Jensen
378 E: objectclass: person
382 E: dn: cn=Bjorn J Jensen, dc=OpenLDAP, dc=org
383 E: cn: Bjorn J Jensen
385 E: objectclass: person
388 E: dn: cn=Jennifer J Jensen, dc=OpenLDAP, dc=org
389 E: cn: Jennifer J Jensen
390 E: cn: Jennifer Jensen
391 E: objectclass: person
393 E: jpegPhoto:: /9j/4AAQSkZJRgABAAAAAQABAAD/2wBDABALD
394 E: A4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQ
395 E: ERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVG
399 Notice that the {{EX: jpegPhoto}} in Jennifer Jensen's entry is
400 encoded using base 64. The {{EX: ldif}} program (described in
401 Section 8.2.6) can be used to produce the LDIF format.
403 Note: Trailing spaces are not trimmed from values in an
404 LDIF file. Nor are multiple internal spaces compressed. If
405 you don't want them in your data, don't put them there.
408 H2: Converting from QUIPU EDB format to LDIF format
410 If you have directory data that is or was held in a QUIPU
411 DSA (available as part of the ISODE package), you will want
412 to convert the EDB files used by QUIPU into an LDIF file.
413 The edb2ldif program is provided to do most of the
414 conversion for you. Once you have an LDIF file, you should
415 follow the steps outlined in section 6.2 above to build an
416 LDBM database for slapd.
420 H3: The {{EX: edb2ldif}} program
422 The edb2ldif program is invoked like this:
424 E: edb2ldif [-d] [-v] [-r] [-o] [-b <basedn>]
425 E: [-a <addvalsfile>] [-f <fileattrdir>]
426 E: [-i <ignoreattr...>] [<edbfile...>]
428 The LDIF data is written to standard output. The arguments
429 have the following meanings:
433 This option enables some debugging output on standard
438 Enable verbose mode that writes status information to
439 standard error, such as which EDB file is being processed,
440 how many entries have been converted so far, etc.
444 Recurse through child directories, processing all EDB files
449 Cause local .add file definitions to override the global addfile
454 Specify the Distinguished Name that all EDB file entries
459 The LDIF information contained in this file will be appended
464 Specify a single directory where all file-based attributes
465 (typically sounds and images) can be found. If this option is
466 not given, file attributes are assumed to be located in the
467 same directory as the EDB file that refers to them.
471 Specify an attribute that should not be converted. You can
472 include as many -i flags as necessary.
476 Specify a particular EDB file (or files) to read data from. By
477 default, the EDB.root (if it exists) and EDB files in the current
480 When {{EX: edb2ldif}} is invoked, it will also look for files named
481 .add in the directories where EDB files are found and append
482 the contents of the .add file to each entry. Typically, this
483 feature is used to include inherited attribute values (e.g.,
484 {{EX: objectClass}}) that do not appear in the EDB files.
488 H3: Step-by-step EDB to LDIF conversion
490 The basic steps to follow when converting your EDB format
491 data to an LDIF file are:
493 ^ Locate the directory at the top of the EDB file hierarchy
494 that your QUIPU DSA masters. The EDB file located there
495 should contain the entries for the first level of your
496 organization or organizational unit. If you are using an
497 indexed database with QUIPU, you may need to create EDB
498 files from your index files (using the synctree or qb2edb
502 + If you do not have a file named EDB.root in the same
503 directory that contains your organizational or organizational
504 unit entry, create it now by hand. Its contents should look
511 .{{EX: objectClass= top & organization & domainRelatedObject &\}}
512 .{{EX: quipuObject & quipuNonLeafObject}}
513 .{{EX: l= Redwood City, California}}
514 .{{EX: st= California}}
515 .{{EX: o=OpenLDAP Project & OpenLDAP Foundation & OpenLDAP}}
516 .{{EX: description=The OpenLDAP Project}}
517 .{{EX: associatedDomain= openldap.org}}
518 .{{EX: masterDSA= c=US@cn=Woolly Monkey}}
521 + (Optional) Create a global add file and/or local .add files to
522 take care of adding any attribute values that do not appear in
523 the EDB files. For example, if all entries in a particular EDB
524 are person entries and you want to add the appropriate
525 objectClass attribute value for them, create a file called .add
526 in the same directory as the person EDB that contains the
529 .{{EX: objectClass: person }}
532 + Run the edb2ldif program to do the actual conversion.
533 Make sure you are in the directory that contains the root of
534 the EDB hierarchy (the one where the EDB.root file resides).
535 Include a -b flag with a base DN one level above your
536 organizational entry, and include -i flags to ignore any
537 attributes that are not useful to slapd. E.g., the command:
539 .{{EX: edb2ldif -v -r -b "c=US" -i iattr -i acl -i xacl -i sacl}}
540 .{{EX: -i lacl -i masterDSA -i slaveDSA > ldif}}
542 will convert the entire EDB hierarchy to LDIF format and
543 write the result to a file named ldif. Some attributes that are
544 not useful when running slapd are ignored. The EDB
545 hierarchy is assumed to reside logically below the base DN
548 + Follow the steps outlined in section 8.2 above to produce
549 an LDBM database from your new LDIF file.
553 H2: The ldbmtest program
555 Occasionally you may find it useful to look at the LDBM
556 database and index files directly (i.e., without going through
557 slapd). The {{EX: ldbmtest}} program is provided for this purpose. It
558 gives you raw access to the database itself. {{EX: ldbmtest}} should
561 E: ldbmtest [-d <debuglevel>] [-f <slapdconfigfile>]
563 The default configuration file in the {{EX: ETCDIR}} is used if you
564 don't supply one. By default, ldbmtest operates on the last
565 database listed in the config file. You can specify an
566 alternate database, or see the current database with the
569 E: b specify an alternate backend database
570 E: B print out the current backend database
572 The {{EX: b}} command will prompt you for the suffix associated with
573 the database you want. The database you select can be
574 viewed and modified using a set of two-letter commands.
575 The first letter selects the command function to perform.
576 Possible commands and their meanings are as follows.
578 E: l lookup (do not follow indirection)
579 E: L lookup (follow indirection)
580 E: t traverse and print keys and data
581 E: T traverse and print keys only
582 E: x delete an index item
583 E: e edit an index item
584 E: a add an index item
585 E: c create an index file
586 E: i insert an entry into an index item
588 The second letter indicates which index the command
589 applies to. The possible index selections are as follows.
591 E: c id2children index
594 E: f arbitrary file name
597 Each command may require additional arguments which
598 ldbmtest will prompt you for.
600 To exit {{EX: ldbmtest}}, type {{EX: control-D}} or {{EX: control-C}}.
602 Note that this is a very raw interface originally developed
603 when testing the database format. It is provided and
604 minimally documented here for interested parties, but it is not
605 meant to be used by the inexperienced. See the next section
606 for a brief description of the LDBM database format.
610 H2: The LDBM database format
612 In normal operation, it is not necessary for you to know much
613 about the LDBM database format. If you are going to use the
614 ldbmtest program to look at or alter the database, or if you
615 want a deeper understanding of how indexes are maintained,
616 some knowledge of how it works could be useful. This
617 section gives an overview of the database format and how
618 slapd makes use of it.
624 The LDBM database works by assigning a compact
625 four-byte unique identifier to each entry in the database. It
626 uses this identifier to refer to entries in indexes. The
627 database consists of one main index file, called id2entry,
628 which maps from an entry's unique identifier (EID) to a text
629 representation of the entry itself. Other index files are
630 maintained, for each indexed attribute for example, that map
631 values people are likely to search on to lists of EIDs.
633 Using this simple scheme, many LDAP queries can be
634 answered efficiently. For example, to answer a search for
635 entries with a surname of "Jensen", slapd would first consult
636 the surname attribute index, look up the value "Jensen" and
637 retrieve the corresponding list of EIDs. Next, slapd would
638 look up each EID in the id2entry index, retrieve the
639 corresponding entry, convert it from text to LDAP format, and
640 return it to the client.
642 The following sections give a very brief overview of each
643 type of index and what it contains. For more detailed
644 information see the paper "An X.500 and LDAP Database:
645 Design and Implementation," available in postscript format
647 {{URL:ftp://terminator.rs.itd.umich.edu/ldap/papers/xldbm.ps}}
650 H3: Attribute index format
652 The LDBM backend will maintain one index file for each
653 attribute it is asked to index. Several sets of keys must
654 coexist in this file (e.g., keys for equality and approximate
655 equality), so the keys are prefixed with a character to ensure
656 uniqueness. The prefixes are given in the table below
659 E: ~ approximate equality keys
660 E: * substring equality keys
661 E: \ continuation keys
663 Key values are also normalized (e.g., converted to upper
664 case for case ignore attributes). So, for example, to look up
665 the surname equality value in the example above using the
666 ldbmtest program, you would look up the value "{{EX: =JENSEN}}".
668 Substring indexes are maintained by generating all possible
669 N-character substrings for a value (N is 3 by default). These
670 substrings are then stored in the attribute index, prefixed by
671 "*". Additional anchors of "^" and "$" are added at the
672 beginning and end of words. So, for example the surname of
673 Jensen would cause the following keys to be entered in the
674 index: {{EX: ^JE, JEN, ENS, NSE, SEN, EN$}}.
676 Approximate values are handled in a similar way, with
677 phonetic codes being generated for each word in a value
678 and then stored in the index, prefixed by "~".
680 Large blocks in the index are split into smaller ones. The
681 smaller blocks are accessed through a level of indirection
682 provided by the original block. They are stored in the index
683 using the continuation key prefix of "\".
689 In addition to the {{EX: id2entry}} and attribute indexes, LDBM
690 maintains a number of other indexes, including the {{EX: dn2id}}
691 index and the {{EX: id2children}} index. These indexes provide the
692 mapping between a DN and the corresponding EID, and the
693 mapping between an EID and the EIDs of the corresponding
694 entry's children, respectively.
696 The {{EX: dn2id}} index stores normalized DNs as keys. The data
697 stored is the corresponding EID.
699 The {{EX: id2children}} index stores EIDs as keys. The data stored
700 is a list of EIDs, just as for the attribute indexes.