I have two LVS boxes, a primary, and a secondary, named lvs1 and lvs2 respectively. Both are gentoo, lvs1 is 64bit, lvs2 is 32bit. Both are running the heartbeat package. Up until I finally discovered the problem, lvs1 was running heartbeat 2.0.8, and lvs2 was running heartbeat 2.0.7-r2. With identical config files for ldirectord.cf, and haresources, etc, healthchecking for ftp resources would hang, and break the rest of healthchecking. Relevant portion of the ldirectord.cf: virtual=10.0.0.195:21 real=192.168.1.195:21 gate 1 real=192.168.1.196:21 gate 1 real=192.168.1.197:21 gate 1 protocol=tcp service=ftp login="testftp" passwd="testftplvs1" persistent=30000 This was identical on both lvs1 and lvs2. When heartbeat was started, and ldirectord started healthchecking and filling in the ipvs table, the virtual service for 10.0.0.195:21 would automatically switch to fallback on lvs1. On lvs2, if heartbeat was stopped on lvs1, the expected behavior occured, and 192.168.1.195, 192.168.1.196, and 192.168.1.197 were added to the pool to be load balanced for the virtual service. Attaching strace to the pid for ldirectord showed that on lvs1, it was only checking 192.168.1.195, 192.168.1.196, and 192.168.1.197, and no other servers. Utilizing lsof against the same pid showed a connection to one of those three ips as well. Performing the same steps (using strace/lsof) on lvs2, again, the expected behavior occured, and connections could be seen to all healthchecked realservers for all virtual services. A reversion to heartbeat 2.0.7-r2 on lvs1 restored the expected behavior, and ftp no longer hangs the healthcheck, nor does the table for that virtual service immediately fail to the fallback. Reproducible: Always Steps to Reproduce: 1. Install heartbeat 2.0.8, utilize above segment of ldirectord.cf, with FTP realservers 2. 3. Actual Results: Healthchecking simply no longer functions, and dead servers (for ANY virtual service) are no longer removed from the pool. FTP virtual service immediately reverts to fallback. Expected Results: Heartbeat/ldirectord function as expected, remove dead servers from pool, and load balance realservers with ftp. As can be seen, this causes problems with any ldirectord.cf containing ftp along with other virtual services.
Thanks for reporting this issue; can you post 'emerge --info' for lvs1 in case it is relevant for reproducing this bug?
lvs1 ~ # emerge --info Portage 2.1.4.5 (default/linux/amd64/2008.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.27-gentoo-r2 x86_64) ================================================================= System uname: 2.6.27-gentoo-r2 x86_64 Intel(R) Pentium(R) D CPU 3.00GHz Timestamp of tree: Tue, 23 Dec 2008 08:30:01 +0000 app-shells/bash: 3.2_p33 dev-lang/python: 2.4.4-r13, 2.5.2-r7 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.61-r2 sys-devel/automake: 1.7.9-r1, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-march=nocona -O2 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LDFLAGS="-Wl,-O1" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://192.168.1.40/gentoo-portage" USE="acl amd64 apache2 berkdb bzip2 cli cracklib crypt cups diskio dri fortran gdbm gpm iconv isdnlog ldirectord mailwrapper midi mmx mudflap multilib ncurses nls nptl nptlonly openmp pam pcre perl pppd python readline reflection sasl session snmp sockets spl sse sse2 ssl sysfs tcpd threads unicode xml xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="fbdev glint i810 intel mach64 mga neomagic nv r128 radeon savage sis tdfx trident vesa vga via vmware voodoo" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
+ 18 Oct 2011; Kacper Kowalik <xarthisius@gentoo.org> -heartbeat-1.2.5.ebuild, + -heartbeat-1.2.5-r1.ebuild, -heartbeat-2.0.7-r2.ebuild, + -files/heartbeat-2.0.7-update-resources-failcount.patch, + -heartbeat-2.0.8.ebuild, -files/heartbeat-2.0.8-asneeded.patch, + -files/heartbeat-2.0.8-crm-assert.patch, + -files/heartbeat-2.0.8-crm-leaks.patch, -files/heartbeat-2.0.8-delay.patch, + -files/heartbeat-2.0.8-glibc.patch, -files/heartbeat-2.0.8-install_fix.patch, + -heartbeat-3.0.4-r1.ebuild: + Drop ancient and unusable versions of heartbeat-{1,2} since we have + heartbeat-3 stable
