Default settings for OpenLDAP recommended backend (bdb) are not very optimal and I seriously think they should be modified. Yesterday I needed to import a 53 MB LDIF file to a fresh installation of OpenLDAP v2.2.28 with bdb backend. Import took around eight hours and I felt something was very wrong, but I didn't have the time to investigate it any further. One bdb file I took my eye on grew about 10 megabytes per hour. Then I tried it again today after seeing this: http://www.openldap.org/faq/data/cache/1072.html So I did put a file called DB_CONFIG to /var/lib/openldap-data/ and made it contain the following lines: --- set_cachesize 0 16777216 0 set_lg_bsize 524288 --- Then I threw away the bdb files from /var/lib/openldap-data and retried the slapadd import. It completed in less than fifteen minutes! The bdb file I watched earlier was growing at least hundreds of kilobytes per second, in bursts about megabyte per second. I retried slapadd again, this time putting slapadd in verbose mode with -v switch. "Adding account..." lines were scrolling past in very fast rate. Then I deleted my DB_CONFIG and retried the import one more time. Whole process became again very sluggish. If you don't want to provide a modified DB_CONFIG, I think at least ebuild should tell a tip about http://www.openldap.org/faq/data/cache/1072.html after installation of OpenLDAP has been completed. This kind of performance difference is huge. I think just growing in-memory cache from 256 kilobytes to 16 megabytes and growing transaction log buffer size from 32 kilobytes to 512 kilobytes shouldn't risk safety. And for in-memory cache 16 megabytes can be an overkill for smaller setups, even less than 16 MB could be enough. Anyway, the default value of 256 kilobytes is too low in my opinion. Reproducible: Always Steps to Reproduce: 1. 2. 3.
And the test system is IBM xSeries 335 with 3.20 GHz P4 Xeon, 1.5 GB of RAM and two U320 SCSI drives in HW RAID-1 mode. File system is ReiserFS v3.6. --- Portage 2.0.51.22-r3 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r2, 2.6.12-gentoo-r4 i686) ================================================================= System uname: 2.6.12-gentoo-r4 i686 Intel(R) Xeon(TM) CPU 3.20GHz Gentoo Base System version 1.6.13 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.15.92.0.2-r10 sys-devel/libtool: 1.5.20 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -mcpu=pentium3 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -mcpu=pentium3 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig buildpkg distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://trumpetti.atm.tut.fi/gentoo/ ftp://mirror.pudas.net/gentoo/" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rotta.mbnet.fi/mbnet-portage" USE="x86 acl aio alsa berkdb big-tables bitmap-fonts cap cluster crypt cscope curl dio eds emboss fortran gdbm gpm gstreamer innodb ipv6 ldap libg++ maildir memlimit mp3 mysql ncurses nethack nls nptl ogg oggvorbis pam pcre perl png posix python readline samba sasl snmp sockets sse ssl tcpd tiff truetype truetype-fonts type1-fonts vorbis zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
btw.: I read an article lately about the switch of the german parliament's servers to Linux, which failed in 2004 and lead to some changes in Samba, so the migration could succeed in 2005. At first they increased the hardcoded number of simultanous connections from 1024 to 8192 clients, but what really helped them - making the former step superfluous - was setting the cache to 100 MB and enabling the idletimeout config option, supporting >5000 users and having 600 simultanous connections on six OpenLDAP servers in the end.
Oh, by the way, I've used set_cachesize parameter successfully in our production servers for more than a year already, so I don't think that's too dangerous. :) The set_lg_bsize is a new discovery for me, but my common sense tells me it should be pretty safe to use, too.
this should be taken care of in openldap 2.3 as it comes with a DB_CONFIG example in both /etc/openldap and /var/lib/openldap-data, hopefully we will have something in portage soon for 2.3. I wouldn't imagine that it is a problem to do it for 2.2 as well, the thing is this really should be a bug filed against bdb as it is with bdb that this should be included, not openldap IMO.
fixed in cvs
Thank you very much!
