>>> Unpacking source... >>> Unpacking ctdb-1.0.73.tgz to /var/tmp/portage/dev-db/ctdb-1.0.73/work >>> Source unpacked in /var/tmp/portage/dev-db/ctdb-1.0.73/work >>> Preparing source in /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73 ... * Running autoheader -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/replace -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/talloc -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/tdb -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/popt -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/events ...  [ ok ] * Running autoconf -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/replace -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/talloc -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/tdb -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/popt -I /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/lib/events ...  [ ok ] >>> Source prepared. >>> Configuring source in /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73 ... * econf: updating ctdb-1.0.73/config.sub with /usr/share/gnuconfig/config.sub * econf: updating ctdb-1.0.73/config.guess with /usr/share/gnuconfig/config.guess * econf: updating ctdb-1.0.73/lib/talloc/config.sub with /usr/share/gnuconfig/config.sub * econf: updating ctdb-1.0.73/lib/talloc/config.guess with /usr/share/gnuconfig/config.guess * econf: updating ctdb-1.0.73/lib/replace/config.sub with /usr/share/gnuconfig/config.sub * econf: updating ctdb-1.0.73/lib/replace/config.guess with /usr/share/gnuconfig/config.guess * econf: updating ctdb-1.0.73/lib/tdb/config.sub with /usr/share/gnuconfig/config.sub * econf: updating ctdb-1.0.73/lib/tdb/config.guess with /usr/share/gnuconfig/config.guess ./configure --prefix=/usr --build=i686-pc-linux-gnu --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib configure: loading site script /usr/share/config.site configure: loading site script /usr/share/crossdev/include/site/linux LIBREPLACE_LOCATION_CHECKS: START checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu LIBREPLACE_LOCATION_CHECKS: END LIBREPLACE_CC_CHECKS: START checking for i686-pc-linux-gnu-gcc... i686-pc-linux-gnu-gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether i686-pc-linux-gnu-gcc accepts -g... yes checking for i686-pc-linux-gnu-gcc option to accept ISO C89... none needed checking for version of gcc... 4.4.0 checking how to run the C preprocessor... i686-pc-linux-gnu-gcc -E checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking minix/config.h usability... no checking minix/config.h presence... no checking for minix/config.h... no checking whether it is safe to define __EXTENSIONS__... yes checking whether byte ordering is bigendian... no checking for inline... inline checking for C99 designated initializers... yes checking for a BSD-compatible install... /usr/bin/install -c checking for library containing strerror... none required checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... 64 checking standards.h usability... no checking standards.h presence... no checking for standards.h... no checking for long long... yes checking for uint_t... no checking for int8_t... yes checking for uint8_t... yes checking for int16_t... yes checking for uint16_t... yes checking for int32_t... yes checking for uint32_t... yes checking for int64_t... yes checking for uint64_t... yes checking for size_t... yes checking for ssize_t... yes checking size of int... 4 checking size of char... 1 checking size of short... 2 checking size of long... 4 checking size of long long... 8 checking size of off_t... 8 checking size of size_t... 4 checking size of ssize_t... 4 checking for intptr_t... yes checking for ptrdiff_t... yes checking for immediate structures... yes LIBREPLACE_CC_CHECKS: END LIBREPLACE_BROKEN_CHECKS: START checking return type of signal handlers... void checking for uid_t in sys/types.h... yes checking for mode_t... yes checking for off_t... yes checking for size_t... (cached) yes checking for pid_t... yes checking for struct stat.st_rdev... yes checking for ino_t... yes checking for loff_t... yes checking for offset_t... no checking for working memcmp... yes checking for pipe... yes checking for strftime... yes checking for srandom... yes checking for random... yes checking for srand... yes checking for rand... yes checking for usleep... yes checking for setbuffer... yes checking for lstat... yes checking for getpgrp... yes checking stdbool.h usability... yes checking stdbool.h presence... yes checking for stdbool.h... yes checking for stdint.h... (cached) yes checking sys/select.h usability... yes checking sys/select.h presence... yes checking for sys/select.h... yes checking setjmp.h usability... yes checking setjmp.h presence... yes checking for setjmp.h... yes checking for stdint.h... (cached) yes checking for stdbool.h... (cached) yes checking for bool... yes checking for _Bool... yes checking for working mmap... no checking sys/syslog.h usability... yes checking sys/syslog.h presence... yes checking for sys/syslog.h... yes checking syslog.h usability... yes checking syslog.h presence... yes checking for syslog.h... yes checking sys/time.h usability... yes checking sys/time.h presence... yes checking for sys/time.h... yes checking time.h usability... yes checking time.h presence... yes checking for time.h... yes checking stdarg.h usability... yes checking stdarg.h presence... yes checking for stdarg.h... yes checking vararg.h usability... no checking vararg.h presence... no checking for vararg.h... no checking sys/socket.h usability... yes checking sys/socket.h presence... yes checking for sys/socket.h... yes checking netinet/in.h usability... yes checking netinet/in.h presence... yes checking for netinet/in.h... yes checking netdb.h usability... yes checking netdb.h presence... yes checking for netdb.h... yes checking arpa/inet.h usability... yes checking arpa/inet.h presence... yes checking for arpa/inet.h... yes checking netinet/ip.h usability... yes checking netinet/ip.h presence... yes checking for netinet/ip.h... yes checking netinet/tcp.h usability... yes checking netinet/tcp.h presence... yes checking for netinet/tcp.h... yes checking netinet/in_systm.h usability... yes checking netinet/in_systm.h presence... yes checking for netinet/in_systm.h... yes checking netinet/in_ip.h usability... no checking netinet/in_ip.h presence... no checking for netinet/in_ip.h... no checking sys/sockio.h usability... no checking sys/sockio.h presence... no checking for sys/sockio.h... no checking sys/un.h usability... yes checking sys/un.h presence... yes checking for sys/un.h... yes checking sys/mount.h usability... yes checking sys/mount.h presence... yes checking for sys/mount.h... yes checking mntent.h usability... yes checking mntent.h presence... yes checking for mntent.h... yes checking for usable net/if.h... yes checking for broken inet_ntoa... no checking for socklen_t... yes checking for sa_family_t... yes checking for struct addrinfo... yes checking for struct sockaddr... yes checking for struct sockaddr_storage... yes checking for struct sockaddr_in6... yes checking for struct sockaddr_storage.ss_family... yes checking for seteuid... yes checking for setresuid... yes checking for setegid... yes checking for setresgid... yes checking for chroot... yes checking for bzero... yes checking for strerror... yes checking for vsyslog... yes checking for setlinebuf... yes checking for mktime... yes checking for ftruncate... yes checking for chsize... no checking for rename... yes checking for waitpid... yes checking for strlcpy... no checking for strlcat... no checking for initgroups... yes checking for memmove... yes checking for strdup... yes checking for pread... yes checking for pwrite... yes checking for strndup... yes checking for strcasestr... yes checking for strtok_r... yes checking for mkdtemp... yes checking for socketpair... yes checking for isatty... yes checking for setresuid declaration... yes checking for setresgid declaration... yes checking for errno declaration... yes checking for secure mkstemp... yes checking stdio.h usability... yes checking stdio.h presence... yes checking for stdio.h... yes checking for strings.h... (cached) yes checking whether snprintf is declared... yes checking whether vsnprintf is declared... yes checking whether asprintf is declared... yes checking whether vasprintf is declared... yes checking for snprintf... yes checking for vsnprintf... yes checking for asprintf... yes checking for vasprintf... yes checking for C99 vsnprintf... yes checking for va_copy... yes checking for __FUNCTION__ macro... yes checking sys/param.h usability... yes checking sys/param.h presence... yes checking for sys/param.h... yes checking limits.h usability... yes checking limits.h presence... yes checking for limits.h... yes checking for comparison_fn_t... yes checking for setenv declaration... yes checking for setenv... yes checking for unsetenv... yes checking for strnlen... yes checking for strtoull... yes checking for __strtoull... no checking for strtouq... yes checking for strtoll... yes checking for __strtoll... no checking for strtoq... yes checking for sig_atomic_t type... yes checking for O_DIRECT flag to open(2)... no checking that the C compiler understands volatile... yes checking for dirent.h that defines DIR... yes checking for library containing opendir... none required checking fcntl.h usability... yes checking fcntl.h presence... yes checking for fcntl.h... yes checking sys/fcntl.h usability... yes checking sys/fcntl.h presence... yes checking for sys/fcntl.h... yes checking sys/resource.h usability... yes checking sys/resource.h presence... yes checking for sys/resource.h... yes checking sys/ioctl.h usability... yes checking sys/ioctl.h presence... yes checking for sys/ioctl.h... yes checking sys/mode.h usability... no checking sys/mode.h presence... no checking for sys/mode.h... no checking sys/filio.h usability... no checking sys/filio.h presence... no checking for sys/filio.h... no checking sys/fs/s5param.h usability... no checking sys/fs/s5param.h presence... no checking for sys/fs/s5param.h... no checking sys/filsys.h usability... no checking sys/filsys.h presence... no checking for sys/filsys.h... no checking sys/acl.h usability... yes checking sys/acl.h presence... yes checking for sys/acl.h... yes checking acl/libacl.h usability... yes checking acl/libacl.h presence... yes checking for acl/libacl.h... yes checking for sys/select.h... (cached) yes checking for sys/time.h... (cached) yes checking utime.h usability... yes checking utime.h presence... yes checking for utime.h... yes checking whether time.h and sys/time.h may both be included... yes checking for sys/wait.h that is POSIX.1 compatible... yes checking sys/capability.h usability... yes checking sys/capability.h presence... yes checking for sys/capability.h... yes checking for broken RedHat 7.2 system header files... no checking for broken RHEL5 sys/capability.h... no checking grp.h usability... yes checking grp.h presence... yes checking for grp.h... yes checking sys/id.h usability... no checking sys/id.h presence... no checking for sys/id.h... no checking compat.h usability... no checking compat.h presence... no checking for compat.h... no checking shadow.h usability... yes checking shadow.h presence... yes checking for shadow.h... yes checking sys/priv.h usability... no checking sys/priv.h presence... no checking for sys/priv.h... no checking for pwd.h... (cached) yes checking sys/security.h usability... no checking sys/security.h presence... no checking for sys/security.h... no checking for getpwnam_r... yes checking for getpwuid_r... yes checking for getpwent_r... yes checking for getpwent_r declaration... yes checking for prototype struct passwd *getpwent_r(struct passwd *src, char *buf, int buflen)... no checking for getgrnam_r... yes checking for getgrgid_r... yes checking for getgrent_r... yes checking for getgrent_r declaration... yes checking for prototype struct group *getgrent_r(struct group *src, char *buf, int buflen)... no checking ctype.h usability... yes checking ctype.h presence... yes checking for ctype.h... yes checking locale.h usability... yes checking locale.h presence... yes checking for locale.h... yes checking fnmatch.h usability... yes checking fnmatch.h presence... yes checking for fnmatch.h... yes checking sys/ipc.h usability... yes checking sys/ipc.h presence... yes checking for sys/ipc.h... yes checking sys/mman.h usability... yes checking sys/mman.h presence... yes checking for sys/mman.h... yes checking sys/shm.h usability... yes checking sys/shm.h presence... yes checking for sys/shm.h... yes checking termios.h usability... yes checking termios.h presence... yes checking for termios.h... yes checking termio.h usability... yes checking termio.h presence... yes checking for termio.h... yes checking sys/termio.h usability... no checking sys/termio.h presence... no checking for sys/termio.h... no checking for library containing dlopen... -ldl checking dlfcn.h usability... yes checking dlfcn.h presence... yes checking for dlfcn.h... yes checking for dlopen... yes checking for dlsym... yes checking for dlerror... yes checking for dlclose... yes checking for library containing shl_load... no checking dl.h usability... no checking dl.h presence... no checking for dl.h... no checking for shl_load... no checking for shl_unload... no checking for shl_findsym... no checking for prototype void *dlopen(const char* filename, unsigned int flags)... no checking whether getpass should be replaced... yes checking whether strptime is available and works... no checking direct.h usability... no checking direct.h presence... no checking for direct.h... no checking windows.h usability... no checking windows.h presence... no checking for windows.h... no checking winsock2.h usability... no checking winsock2.h presence... no checking for winsock2.h... no checking ws2tcpip.h usability... no checking ws2tcpip.h presence... no checking for ws2tcpip.h... no checking whether mkdir supports mode... yes checking for timegm... yes checking for inet_ntop... yes checking for inet_pton... yes checking for getaddrinfo... yes checking for broken readdir... yes checking for getdirentries... yes checking for prototype long telldir(const DIR *dir)... no checking for prototype int seekdir(DIR *dir, long ofs)... no checking for replacing readdir using getdirentries()... no checking for getdents... no checking for replacing readdir using getdents()... no checking a usable readdir()... no configure: WARNING: the provided readdir() is broken checking ifaddrs.h usability... yes checking ifaddrs.h presence... yes checking for ifaddrs.h... yes checking for struct sockaddr.sa_len... no checking for getifaddrs and freeifaddrs... yes checking for iface getifaddrs... lo eth0 eth1 wmaster0 wlan0 vde0 pan0 lo IP=127.0.0.1 NETMASK=255.0.0.0 eth0 IP=192.168.0.17 NETMASK=255.255.255.240 eth0:1 IP=192.168.0.19 NETMASK=255.255.255.240 eth1 IP=192.168.0.33 NETMASK=255.255.255.240 eth1:1 IP=192.168.28.1 NETMASK=255.255.255.0 wlan0 IP=172.16.1.11 NETMASK=255.255.0.0 wlan0:1 IP=172.16.0.2 NETMASK=255.255.0.0 vde0 IP=192.168.0.49 NETMASK=255.255.255.240 yes checking for syslog... yes checking for printf... yes checking for memset... yes checking for memcpy... yes LIBREPLACE_BROKEN_CHECKS: END checking popt.h usability... yes checking popt.h presence... yes checking for popt.h... yes checking for poptGetContext in -lpopt... yes checking whether to use included popt... no checking size of size_t... (cached) 4 checking size of void *... 4 checking for mmap... yes checking for pread... (cached) yes checking for pwrite... (cached) yes checking for getpagesize... yes checking for utime... yes checking getopt.h usability... yes checking getopt.h presence... yes checking for getopt.h... yes checking for sys/select.h... (cached) yes checking for sys/time.h... (cached) yes checking for pread declaration... yes checking for pwrite declaration... yes checking sys/epoll.h usability... yes checking sys/epoll.h presence... yes checking for sys/epoll.h... yes checking for epoll_create... yes checking libipq.h usability... yes checking libipq.h presence... yes checking for libipq.h... yes checking linux/netfilter.h usability... no checking linux/netfilter.h presence... yes configure: WARNING: linux/netfilter.h: present but cannot be compiled configure: WARNING: linux/netfilter.h: check for missing prerequisite headers? configure: WARNING: linux/netfilter.h: see the Autoconf documentation configure: WARNING: linux/netfilter.h: section "Present But Cannot Be Compiled" configure: WARNING: linux/netfilter.h: proceeding with the preprocessor's result configure: WARNING: linux/netfilter.h: in the future, the compiler will take precedence checking for linux/netfilter.h... yes checking sched.h usability... yes checking sched.h presence... yes checking for sched.h... yes checking for sched_setscheduler... yes checking for sin_len in sock... no configure: creating ./config.status config.status: creating Makefile config.status: creating config.h config.status: executing rm-stdint.h commands config.status: executing rm-stdbool.h commands >>> Source configured. >>> Compiling source in /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73 ... make -j14 ctdb will be compiled with flags: CFLAGS = -g -I./include -Iinclude -Ilib -Ilib/util -I. -I./lib/talloc -I./lib/tdb/include -I./lib/replace -DVARDIR=\"/var/lib\" -DETCDIR=\"/etc\" -DLOGDIR=\"/var/lib/log\" -DUSE_MMAP=1 -O2 -pipe -I./lib/replace -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings Compiling server/ctdbd.c LIBS = Compiling server/ctdb_daemon.c Compiling server/ctdb_lockwait.c Compiling server/ctdb_recoverd.c Compiling server/ctdb_recover.c Compiling server/ctdb_freeze.c Compiling server/ctdb_tunables.c Compiling server/ctdb_monitor.c Compiling server/ctdb_server.c server/ctdb_lockwait.c: In function 'ctdb_lockwait': server/ctdb_lockwait.c:140: warning: ignoring return value of 'write', declared with attribute warn_unused_result Compiling server/ctdb_control.c Compiling server/ctdb_call.c server/ctdb_freeze.c: In function 'ctdb_freeze_lock': server/ctdb_freeze.c:189: warning: ignoring return value of 'read', declared with attribute warn_unused_result Compiling server/ctdb_ltdb_server.c Compiling server/ctdb_traverse.c Compiling server/eventscript.c Compiling server/ctdb_takeover.c Compiling server/ctdb_serverids.c Compiling server/ctdb_persistent.c Compiling server/ctdb_keepalive.c Compiling server/ctdb_logging.c Compiling client/ctdb_client.c Compiling common/ctdb_io.c server/ctdb_recover.c: In function 'ctdb_control_set_recmode': server/ctdb_recover.c:667: warning: ignoring return value of 'write', declared with attribute warn_unused_result server/ctdb_logging.c: In function 'ctdb_logfile_log': server/ctdb_logging.c:85: warning: ignoring return value of 'vasprintf', declared with attribute warn_unused_result server/ctdb_logging.c:92: warning: ignoring return value of 'asprintf', declared with attribute warn_unused_result server/ctdb_logging.c:96: warning: ignoring return value of 'write', declared with attribute warn_unused_result Compiling common/ctdb_util.c Compiling common/ctdb_ltdb.c Compiling common/ctdb_message.c Compiling common/cmdline.c server/eventscript.c: In function 'ctdb_event_script_handler': server/eventscript.c:214: warning: ignoring return value of 'read', declared with attribute warn_unused_result server/eventscript.c: In function 'ctdb_run_eventscripts': server/eventscript.c:498: warning: format not a string literal and no format arguments Compiling lib/util/debug.c Compiling common/rb_tree.c Compiling common/system_linux.c Compiling lib/util/idtree.c server/ctdb_persistent.c: In function 'ctdb_childwrite': server/ctdb_persistent.c:407: warning: ignoring return value of 'write', declared with attribute warn_unused_result Compiling lib/util/db_wrap.c lib/util/debug.c: In function '_do_debug_v': lib/util/debug.c:32: warning: ignoring return value of 'vasprintf', declared with attribute warn_unused_result Compiling lib/util/strlist.c common/ctdb_util.c: In function 'parse_ip_port': common/ctdb_util.c:438: warning: passing argument 3 of 'parse_ip' from incompatible pointer type common/../include/ctdb_private.h:1286: note: expected 'union ctdb_sock_addr *' but argument is of type 'const char *' Compiling lib/util/util.c Compiling lib/util/util_time.c Compiling lib/util/util_file.c Compiling talloc.c Compiling common/tdb.c Compiling common/dump.c server/ctdb_takeover.c: In function 'ctdb_control_tcp_client': server/ctdb_takeover.c:1057: warning: format '%lu' expects type 'long unsigned int', but argument 3 has type 'unsigned int' server/ctdb_takeover.c:1057: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'unsigned int' Compiling common/transaction.c Compiling common/error.c Compiling common/traverse.c Compiling common/freelist.c Compiling common/freelistcheck.c Compiling common/io.c Compiling common/lock.c Compiling common/open.c Compiling replace.c Compiling snprintf.c Compiling getpass.c Compiling strptime.c Compiling lib/events/events.c Compiling lib/events/events_select.c ./lib/replace/getpass.c: In function 'rep_getpass': ./lib/replace/getpass.c:188: warning: ignoring return value of 'fgets', declared with attribute warn_unused_result Compiling lib/events/events_signal.c Compiling lib/events/events_timed.c Compiling lib/events/events_standard.c Compiling lib/events/events_epoll.c Compiling tcp/tcp_connect.c Compiling tcp/tcp_io.c Compiling tcp/tcp_init.c Compiling tools/ctdb.c Compiling tools/ctdb_vacuum.c Compiling utils/scsi_io/scsi_io.c Compiling utils/ipmux/ipmux.c Compiling tests/src/ctdb_bench.c Generating utils/smnotify/smnotify.h rpcgen -h utils/smnotify/smnotify.x > utils/smnotify/smnotify.h Compiling tests/src/ctdb_fetch.c Compiling tests/src/ctdb_store.c Compiling tests/src/ctdb_randrec.c Compiling tests/src/ctdb_persistent.c lib/events/events_signal.c: In function 'signal_handler': lib/events/events_signal.c:81: warning: ignoring return value of 'write', declared with attribute warn_unused_result lib/events/events_signal.c: In function 'signal_pipe_handler': lib/events/events_signal.c:137: warning: ignoring return value of 'read', declared with attribute warn_unused_result lib/events/events_signal.c: In function 'common_event_add_signal': lib/events/events_signal.c:219: warning: ignoring return value of 'pipe', declared with attribute warn_unused_result tcp/tcp_init.c: In function 'tnode_destructor': tcp/tcp_init.c:30: warning: unused variable 'node' Compiling tests/src/ctdb_traverse.c Compiling tests/src/rb_test.c Compiling tests/src/ctdb_transaction.c Generating utils/smnotify/gen_xdr.c rpcgen -c utils/smnotify/smnotify.x > utils/smnotify/gen_xdr.c Generating utils/smnotify/gen_smnotify.c rpcgen -l utils/smnotify/smnotify.x > utils/smnotify/gen_smnotify.c Compiling utils/smnotify/smnotify.c Linking bin/ctdbd Linking tests/bin/ctdb_bench Linking bin/ctdb_ipmux Compiling utils/smnotify/gen_xdr.c Compiling utils/smnotify/gen_smnotify.c Linking tests/bin/ctdb_store Linking tests/bin/ctdb_randrec Linking tests/bin/ctdb_fetch Linking tests/bin/ctdb_traverse Linking tests/bin/ctdb_transaction Linking tests/bin/ctdb_persistent utils/smnotify/gen_xdr.c: In function 'xdr_status': utils/smnotify/gen_xdr.c:11: warning: unused variable 'buf' Linking bin/scsi_io Linking bin/smnotify Linking tests/bin/rb_test tools/ctdb.c: In function 'control_backupdb': tools/ctdb.c:2220: warning: ignoring return value of 'write', declared with attribute warn_unused_result tools/ctdb.c:2221: warning: ignoring return value of 'write', declared with attribute warn_unused_result tools/ctdb.c: In function 'control_restoredb': tools/ctdb.c:2260: warning: ignoring return value of 'read', declared with attribute warn_unused_result tools/ctdb.c:2275: warning: ignoring return value of 'read', declared with attribute warn_unused_result tools/ctdb.c: In function 'control_dumpmemory': tools/ctdb.c:2462: warning: ignoring return value of 'write', declared with attribute warn_unused_result tools/ctdb.c: In function 'mem_dump_handler': tools/ctdb.c:2473: warning: ignoring return value of 'write', declared with attribute warn_unused_result Linking bin/ctdb >>> Source compiled. >>> Test phase [test]: dev-db/ctdb-1.0.73 make -j14 -j1 test ctdb will be compiled with flags: CFLAGS = -g -I./include -Iinclude -Ilib -Ilib/util -I. -I./lib/talloc -I./lib/tdb/include -I./lib/replace -DVARDIR=\"/var/lib\" -DETCDIR=\"/etc\" -DLOGDIR=\"/var/lib/log\" -DUSE_MMAP=1 -O2 -pipe -I./lib/replace -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings LIBS = tests/run_tests.sh --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/00_ctdb_init.sh (23:44:20) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Restarting ctdb on all nodes... Attempting to politely shutdown daemons... common/cmdline.c:134 Failed to connect to daemon 2009/07/16 23:44:20.224306 [22254]: Failed to init ctdb Sleeping for a while... =1|.| Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/00_ctdb_init.sh ========================================================================== TEST PASSED: tests/simple/00_ctdb_init.sh (duration: 22s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/00_ctdb_install_eventscript.sh (23:44:42) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- ALL OK: tests/simple/00_ctdb_install_eventscript.sh ========================================================================== TEST PASSED: tests/simple/00_ctdb_install_eventscript.sh (duration: 0s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/00_ctdb_onnode.sh (23:44:42) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Checking connectivity between nodes... >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.1 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.2 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.1 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.1 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.2 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.2 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.1 << >> NODE: /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.2 << Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/00_ctdb_onnode.sh ========================================================================== TEST PASSED: tests/simple/00_ctdb_onnode.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/01_ctdb_version.sh (23:44:44) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Failed to execute "rpm -q ctdb" on node(s) "0" error: cannot open Name index using db3 - No such file or directory (2) package ctdb is not installed No useful output from rpm, SKIPPING rest of test. Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/01_ctdb_version.sh ========================================================================== TEST PASSED: tests/simple/01_ctdb_version.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/02_ctdb_listvars.sh (23:44:47) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb listvars": MaxRedirectCount = 3 SeqnumFrequency = 1 ControlTimeout = 60 TraverseTimeout = 20 KeepaliveInterval = 5 KeepaliveLimit = 5 MaxLACount = 7 RecoverTimeout = 30 RecoverInterval = 1 ElectionTimeout = 3 TakeoverTimeout = 5 MonitorInterval = 15 TickleUpdateInterval = 20 EventScriptTimeout = 20 EventScriptBanCount = 5 RecoveryGracePeriod = 60 RecoveryBanPeriod = 300 DatabaseHashSize = 10000 DatabaseMaxDead = 5 RerecoveryTimeout = 1 EnableBans = 1 DeterministicIPs = 1 DisableWhenUnhealthy = 0 ReclockPingPeriod = 60 NoIPFailback = 0 VerboseMemoryNames = 0 RecdPingTimeout = 20 RecdFailCount = 3 LogLatencyMs = 0 There are 29 lines of output Output lines look OK Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/02_ctdb_listvars.sh ========================================================================== TEST PASSED: tests/simple/02_ctdb_listvars.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/03_ctdb_getvar.sh (23:44:49) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb listvars": MaxRedirectCount = 3 SeqnumFrequency = 1 ControlTimeout = 60 TraverseTimeout = 20 KeepaliveInterval = 5 KeepaliveLimit = 5 MaxLACount = 7 RecoverTimeout = 30 RecoverInterval = 1 ElectionTimeout = 3 TakeoverTimeout = 5 MonitorInterval = 15 TickleUpdateInterval = 20 EventScriptTimeout = 20 EventScriptBanCount = 5 RecoveryGracePeriod = 60 RecoveryBanPeriod = 300 DatabaseHashSize = 10000 DatabaseMaxDead = 5 RerecoveryTimeout = 1 EnableBans = 1 DeterministicIPs = 1 DisableWhenUnhealthy = 0 ReclockPingPeriod = 60 NoIPFailback = 0 VerboseMemoryNames = 0 RecdPingTimeout = 20 RecdFailCount = 3 LogLatencyMs = 0 Veryifying all variable values using "ctdb getvar"... Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/03_ctdb_getvar.sh ========================================================================== TEST PASSED: tests/simple/03_ctdb_getvar.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/04_ctdb_setvar.sh (23:44:51) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getvar RecoverTimeout": RecoverTimeout = 30 Going to try incrementing it... That seemed to work, let's check the value... Output of "ctdb getvar RecoverTimeout": RecoverTimeout = 31 Look's good! Now verifying with "ctdb listvars" Output of "ctdb listvars | grep '^RecoverTimeout'": RecoverTimeout = 31 Look's good! Putting the old value back... All done... Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/04_ctdb_setvar.sh ========================================================================== TEST PASSED: tests/simple/04_ctdb_setvar.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/05_ctdb_listnodes.sh (23:44:54) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb listnodes": 127.0.0.1 127.0.0.2 127.0.0.3 There are 3 lines of output Output lines look OK Checking other nodes... Node 1: OK Node 2: OK Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/05_ctdb_listnodes.sh ========================================================================== TEST PASSED: tests/simple/05_ctdb_listnodes.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/06_ctdb_getpid.sh (23:44:57) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY There are 3 nodes... Output of "onnode -q all ctdb getpid": Pid:22262 Pid:22267 Pid:22272 Output of "ctdb getpid -n all": Pid:22262 Pid:22267 Pid:22272 Output of "( ctdb getpid -n 0; ctdb getpid -n 1; ctdb getpid -n 2 )": Pid:22262 Pid:22267 Pid:22272 They're the same... cool! Checking each PID for validity Node 0, PID 22262 looks to be running "ctdbd" - GOOD! Node 1, PID 22267 looks to be running "ctdbd" - GOOD! Node 2, PID 22272 looks to be running "ctdbd" - GOOD! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/06_ctdb_getpid.sh ========================================================================== TEST PASSED: tests/simple/06_ctdb_getpid.sh (duration: 5s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/07_ctdb_process_exists.sh (23:45:02) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Checking for PID 24215 on node 1 1:24215 exists 24215 OK Checking for PID 24232 on node 1 1:24232 does not exist 24232 OK Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/07_ctdb_process_exists.sh ========================================================================== TEST PASSED: tests/simple/07_ctdb_process_exists.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/08_ctdb_isnotrecmaster.sh (23:45:05) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb isnotrecmaster || true": this node is not the recmaster this node is the recmaster this node is not the recmaster OK, there is only 1 recmaster OK, all the other nodes claim not to be the recmaster Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/08_ctdb_isnotrecmaster.sh ========================================================================== TEST PASSED: tests/simple/08_ctdb_isnotrecmaster.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/09_ctdb_ping.sh (23:45:07) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb ping -n 1": response from 1 time=0.000056 sec (1 clients) There are 1 lines of output Output lines look OK Waiting until node 1 has status "disconnected"... <30|| OK Output of "! ctdb ping -n 1": 2009/07/16 23:45:07.321861 [24611]: Node 1 is DISCONNECTED There are 1 lines of output Output lines look OK Expect a restart... Cluster is UNHEALTHY Restarting ctdb on all nodes to get back into known state... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready ALL OK: tests/simple/09_ctdb_ping.sh ========================================================================== TEST PASSED: tests/simple/09_ctdb_ping.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/11_ctdb_ip.sh (23:45:30) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting list of public IPs... Output of "ctdb ip -n all": Public IPs on ALL nodes 192.0.2.6 2 192.0.2.5 1 192.0.2.3 2 192.0.2.2 1 GOOD: node 2 appears to have 192.0.2.6 assigned GOOD: node 1 appears to have 192.0.2.5 assigned GOOD: node 2 appears to have 192.0.2.3 assigned GOOD: node 1 appears to have 192.0.2.2 assigned Looks good! Checking that "ctdb -Y ip -n all | sed -e '1d'" produces expected output... Yep, looks good! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/11_ctdb_ip.sh ========================================================================== TEST PASSED: tests/simple/11_ctdb_ip.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/12_ctdb_getdebug.sh (23:45:32) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "onnode -q all ctdb getdebug": Node 0 is at debug level ERR (0) Node 1 is at debug level ERR (0) Node 2 is at debug level ERR (0) There are 3 lines of output Output lines look OK Output of "ctdb getdebug -n all": Node 0 is at debug level ERR (0) Node 1 is at debug level ERR (0) Node 2 is at debug level ERR (0) Output of "ctdb getdebug -n 0; ctdb getdebug -n 1; ctdb getdebug -n 2": Node 0 is at debug level ERR (0) Node 1 is at debug level ERR (0) Node 2 is at debug level ERR (0) They're the same... cool! Checking that "ctdb -Y getdebug -n all" produces expected output... Yep, looks good! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/12_ctdb_getdebug.sh ========================================================================== TEST PASSED: tests/simple/12_ctdb_getdebug.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/13_ctdb_setdebug.sh (23:45:35) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getdebug": Node 1 is at debug level ERR (0) Setting debug level on node 1 to EMERG. Output of "ctdb getdebug": Node 1 is at debug level EMERG (-3) That seemed to work... cool! Returning the debug level to its initial value... Setting debug level on node 1 to ERR. Output of "ctdb getdebug": Node 1 is at debug level ERR (0) That seemed to work... cool! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/13_ctdb_setdebug.sh ========================================================================== TEST PASSED: tests/simple/13_ctdb_setdebug.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/14_ctdb_statistics.sh (23:45:37) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb statistics": CTDB version 1 num_clients 2 frozen 0 recovering 0 client_packets_sent 389 client_packets_recv 385 node_packets_sent 865 node_packets_recv 167 keepalive_packets_sent 8 keepalive_packets_recv 8 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 8 req_control 471 reply_control 383 client req_call 0 req_message 1 req_control 384 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 10073 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec There are 38 lines of output Output lines look OK Output of "ctdb statistics -n all": Gathered statistics for 3 nodes CTDB version 1 num_clients 4 frozen 0 recovering 0 client_packets_sent 1302 client_packets_recv 1293 node_packets_sent 2625 node_packets_recv 623 keepalive_packets_sent 24 keepalive_packets_recv 24 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 24 req_control 1295 reply_control 1284 client req_call 0 req_message 8 req_control 1285 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 27435 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec There are 39 lines of output Output lines look OK Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/14_ctdb_statistics.sh ========================================================================== TEST PASSED: tests/simple/14_ctdb_statistics.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/15_ctdb_statisticsreset.sh (23:45:39) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting initial statistics for node 0... Output of "ctdb statistics": CTDB version 1 num_clients 2 frozen 0 recovering 0 client_packets_sent 729 client_packets_recv 732 node_packets_sent 1256 node_packets_recv 341 keepalive_packets_sent 8 keepalive_packets_recv 8 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 9 req_control 501 reply_control 723 client req_call 0 req_message 8 req_control 724 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 8173 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec Output of "ctdb statistics": CTDB version 1 num_clients 0 frozen 0 recovering 0 client_packets_sent 3 client_packets_recv 3 node_packets_sent 6 node_packets_recv 0 keepalive_packets_sent 0 keepalive_packets_recv 0 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 0 req_control 3 reply_control 3 client req_call 0 req_message 0 req_control 3 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 8173 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec GOOD: req_control reduced from 501 to 3 GOOD: reply_control reduced from 723 to 3 GOOD: node_packets_recv reduced from 341 to 0 Getting initial statistics for node 1... Output of "ctdb statistics": CTDB version 1 num_clients 2 frozen 0 recovering 0 client_packets_sent 434 client_packets_recv 430 node_packets_sent 974 node_packets_recv 195 keepalive_packets_sent 8 keepalive_packets_recv 8 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 9 req_control 535 reply_control 428 client req_call 0 req_message 1 req_control 429 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 10369 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec Output of "ctdb statistics": CTDB version 1 num_clients 0 frozen 0 recovering 0 client_packets_sent 3 client_packets_recv 3 node_packets_sent 6 node_packets_recv 0 keepalive_packets_sent 0 keepalive_packets_recv 0 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 0 req_control 3 reply_control 3 client req_call 0 req_message 0 req_control 3 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 10369 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec GOOD: req_control reduced from 535 to 3 GOOD: reply_control reduced from 428 to 3 GOOD: node_packets_recv reduced from 195 to 0 Getting initial statistics for node 2... Output of "ctdb statistics": CTDB version 1 num_clients 2 frozen 0 recovering 0 client_packets_sent 336 client_packets_recv 331 node_packets_sent 795 node_packets_recv 190 keepalive_packets_sent 8 keepalive_packets_recv 8 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 9 req_control 457 reply_control 330 client req_call 0 req_message 0 req_control 331 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 10369 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec Output of "ctdb statistics": CTDB version 1 num_clients 0 frozen 0 recovering 0 client_packets_sent 3 client_packets_recv 3 node_packets_sent 6 node_packets_recv 0 keepalive_packets_sent 0 keepalive_packets_recv 0 node req_call 0 reply_call 0 req_dmaster 0 reply_dmaster 0 reply_error 0 req_message 0 req_control 3 reply_control 3 client req_call 0 req_message 0 req_control 3 timeouts call 0 control 0 traverse 0 total_calls 0 pending_calls 0 lockwait_calls 0 pending_lockwait_calls 0 childwrite_calls 0 pending_childwrite_calls 0 memory_used 10369 max_hop_count 0 max_call_latency 0.000000 sec max_lockwait_latency 0.000000 sec max_childwrite_latency 0.000000 sec GOOD: req_control reduced from 457 to 3 GOOD: reply_control reduced from 330 to 3 GOOD: node_packets_recv reduced from 190 to 0 Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/15_ctdb_statisticsreset.sh ========================================================================== TEST PASSED: tests/simple/15_ctdb_statisticsreset.sh (duration: 3s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/16_ctdb_config_add_ip.sh (23:45:42) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting list of public IPs... Selected node 2 with IPs: 192.0.2.6 192.0.2.3 Adding IP: 192.0.2.1/24 on interface lo Waiting for IP to be added... <60|| OK That worked! Restarting cluster to restore configuration... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/16_ctdb_config_add_ip.sh ========================================================================== TEST PASSED: tests/simple/16_ctdb_config_add_ip.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/17_ctdb_config_delete_ip.sh (23:46:05) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting list of public IPs... Output of "ctdb ip -n all | sed -e '1d'": 192.0.2.6 2 192.0.2.5 1 192.0.2.3 2 192.0.2.2 1 Attempting to remove 192.0.2.2 from node 1. Sleeping... =1|.| That worked! Restarting cluster to restore configuration... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/17_ctdb_config_delete_ip.sh ========================================================================== TEST PASSED: tests/simple/17_ctdb_config_delete_ip.sh (duration: 25s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/18_ctdb_freeze.sh (23:46:30) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Freezing node 1 Waiting until node 1 has status "frozen"... <30|| OK That worked! Restarting cluster to restore configuration... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/18_ctdb_freeze.sh ========================================================================== TEST PASSED: tests/simple/18_ctdb_freeze.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/19_ctdb_thaw.sh (23:46:53) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Freezing node 1 Waiting until node 1 has status "frozen"... <30|| OK That worked! Now thawing node 1 Waiting until node 1 has status "unfrozen"... <30|| OK Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/19_ctdb_thaw.sh ========================================================================== TEST PASSED: tests/simple/19_ctdb_thaw.sh (duration: 1s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/20_ctdb_getmonmode.sh (23:46:54) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getmonmode -n 1": Monitoring mode:ACTIVE (0) There are 1 lines of output Output lines look OK Output of "ctdb -Y getmonmode -n 1": :mode: :0: Looks OK Output of "ctdb disablemonitor -n 1": Monitoring mode:DISABLED Waiting until node 1 has status "monoff"... <30|| OK That worked! Restarting cluster to restore configuration... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/20_ctdb_getmonmode.sh ========================================================================== TEST PASSED: tests/simple/20_ctdb_getmonmode.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/21_ctdb_disablemonitor.sh (23:47:17) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getvar MonitorInterval -n 1": MonitorInterval = 15 Monitor interval on node 1 is 15 seconds. Creating trigger file on node 1 to make it unhealthy... Waiting until node 1 has status "unhealthy"... <15|....| OK Output of "ls -l /tmp/ctdb-test-unhealthy-detected.1": -rw-r--r-- 1 root root 0 Jul 16 23:47 /tmp/ctdb-test-unhealthy-detected.1 Waiting until recovery is complete... <30|.| OK Output of "ctdb disablemonitor -n 1": Monitoring mode:DISABLED There are 1 lines of output Output lines look OK Waiting until node 1 has status "monoff"... <30|| OK Output of "rm -v /tmp/ctdb-test-unhealthy-detected.1": removed `/tmp/ctdb-test-unhealthy-detected.1' =15|...............| OK: flag file was not recreated so monitoring must be disabled. Removing trigger file. Monitoring is disabled so node will stay unhealthy. Output of "rm -v /tmp/ctdb-test-unhealthy-trigger.1": removed `/tmp/ctdb-test-unhealthy-trigger.1' =15|...............| OK, that all worked. Expect a restart... Cluster is UNHEALTHY Restarting ctdb on all nodes to get back into known state... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready ALL OK: tests/simple/21_ctdb_disablemonitor.sh ========================================================================== TEST PASSED: tests/simple/21_ctdb_disablemonitor.sh (duration: 57s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/22_ctdb_enablemonitor.sh (23:48:14) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getvar MonitorInterval -n 1": MonitorInterval = 15 Monitor interval on node 1 is 15 seconds. Output of "ctdb disablemonitor -n 1": Monitoring mode:DISABLED There are 1 lines of output Output lines look OK Waiting until node 1 has status "monoff"... <30|| OK Creating trigger file on node 1 to see if it goes unhealthy... =15|...............| OK: flag file was not created so monitoring must be disabled. Output of "ctdb enablemonitor -n 1": Monitoring mode:ACTIVE There are 1 lines of output Output lines look OK Waiting until node 1 has status "monon"... <30|| OK =15|...............| OK: flag file was created so monitoring must be enabled. Waiting until node 1 has status "unhealthy"... <15|| OK Output of "ls -l /tmp/ctdb-test-unhealthy-detected.1": -rw-r--r-- 1 root root 0 Jul 16 23:48 /tmp/ctdb-test-unhealthy-detected.1 OK, that all worked. Expect a restart... removed `/tmp/ctdb-test-unhealthy-trigger.1' Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/22_ctdb_enablemonitor.sh ========================================================================== TEST PASSED: tests/simple/22_ctdb_enablemonitor.sh (duration: 54s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/23_ctdb_moveip.sh (23:49:08) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY There are 3 nodes... Getting list of public IPs... Output of "ctdb ip -n all | sed -e '1d'": 192.0.2.6 2 192.0.2.5 1 192.0.2.3 2 192.0.2.2 1 Sanity checking IPs... OK Turning off DeterministicIPs... Turning on NoIPFailback... Attempting to move 192.0.2.2 from node 1 to node 2. Waiting for IPs to fail over... <60|| OK IP moved from 1. Waiting for IPs to fail over... <60|| OK IP moved to 2. OK, that worked... expect a restart... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/23_ctdb_moveip.sh ========================================================================== TEST PASSED: tests/simple/23_ctdb_moveip.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/24_ctdb_getdbmap.sh (23:49:31) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb getdbmap": Number of databases:0 There are 1 lines of output Output lines look OK Creating test database: 48091806b93c907fd2525173474df62e.tdb There are 2 lines of output Output lines look OK OK: correct number of additional databases OK: getdbmap knows about "48091806b93c907fd2525173474df62e.tdb" Creating test database: 35abe2f4033eddccde1354fc53ad1ec8.tdb There are 3 lines of output Output lines look OK OK: correct number of additional databases OK: getdbmap knows about "35abe2f4033eddccde1354fc53ad1ec8.tdb" Creating test database: 7085c872d508414c5b8cbf608e101dbb.tdb There are 4 lines of output Output lines look OK OK: correct number of additional databases OK: getdbmap knows about "7085c872d508414c5b8cbf608e101dbb.tdb" Creating test database: 14d26555de8be11224bed29d832c96b9.tdb There are 5 lines of output Output lines look OK OK: correct number of additional databases OK: getdbmap knows about "14d26555de8be11224bed29d832c96b9.tdb" Creating test database: 985f77265f2aa29acc41fa0705a4a371.tdb There are 6 lines of output Output lines look OK OK: correct number of additional databases OK: getdbmap knows about "985f77265f2aa29acc41fa0705a4a371.tdb" OK, that worked... expect a restart... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/24_ctdb_getdbmap.sh ========================================================================== TEST PASSED: tests/simple/24_ctdb_getdbmap.sh (duration: 23s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/25_dumpmemory.sh (23:49:54) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Output of "ctdb dumpmemory": full talloc report on 'null_context' (total 8173 bytes in 115 blocks) struct event_context contains 84 bytes in 3 blocks (ref 0) 0x8c8de98 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91bf0 struct std_event_context contains 28 bytes in 1 blocks (ref 0) 0x8c8dee0 struct event_context contains 5947 bytes in 101 blocks (ref 0) 0x8c8d9e0 struct ctdb_context contains 5899 bytes in 99 blocks (ref 0) 0x8c8da78 talloc_new: server/ctdb_server.c:243 contains 68 bytes in 3 blocks (ref 0) 0x8c925d0 server/ctdb_server.c:418 contains 68 bytes in 2 blocks (ref 0) 0x8c90f48 TDB_DATA contains 8 bytes in 1 blocks (ref 0) 0x8c90238 struct queue_next contains 40 bytes in 2 blocks (ref 0) 0x8c923c0 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c92538 struct ctdb_client contains 270 bytes in 8 blocks (ref 0) 0x8c92370 struct daemon_control_state contains 140 bytes in 4 blocks (ref 0) 0x8c90d98 struct ctdb_control_state contains 52 bytes in 2 blocks (ref 0) 0x8c920e8 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c90fb8 common/ctdb_io.c:77 contains 64 bytes in 1 blocks (ref 0) 0x8c924c0 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c91328 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91138 struct ctdb_client: pid:31211 contains 30 bytes in 1 blocks (ref 0) 0x8c927d8 struct ctdb_vnn_map contains 24 bytes in 2 blocks (ref 0) 0x8c8fa20 uint32_t contains 12 bytes in 1 blocks (ref 0) 0x8c8fa60 uint32_t contains 36 bytes in 2 blocks (ref 0) 0x8c912f0 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c92658 struct ctdb_client contains 326 bytes in 11 blocks (ref 0) 0x8c90e48 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c92320 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c922d0 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c92280 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c92230 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c921e0 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c92190 struct ctdb_message_list contains 28 bytes in 1 blocks (ref 0) 0x8c92140 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c90ef0 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91d08 struct ctdb_client: pid:30772 contains 30 bytes in 1 blocks (ref 0) 0x8c90e98 struct signal_event contains 40 bytes in 2 blocks (ref 0) 0x8c91ae0 reference to: struct sig_state talloc_new: server/ctdb_takeover.c:1928 contains 32 bytes in 2 blocks (ref 0) 0x8c91260 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c92780 struct ctdb_monitor_state contains 44 bytes in 3 blocks (ref 0) 0x8c91190 talloc_new: server/ctdb_monitor.c:197 contains 32 bytes in 2 blocks (ref 0) 0x8c911d0 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c92400 talloc_new: server/ctdb_keepalive.c:87 contains 32 bytes in 2 blocks (ref 0) 0x8c91100 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c91298 struct timed_event contains 32 bytes in 1 blocks (ref 0) 0x8c90df0 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c902e0 127.0.0.1:4379 contains 15 bytes in 1 blocks (ref 0) 0x8c8e9c0 127.0.0.1 contains 10 bytes in 1 blocks (ref 0) 0x8c8dfe0 struct ctdb_tcp contains 212 bytes in 8 blocks (ref 0) 0x8c8df88 struct ctdb_incoming contains 84 bytes in 3 blocks (ref 0) 0x8c91ec8 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c91e50 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91f08 struct ctdb_incoming contains 84 bytes in 3 blocks (ref 0) 0x8c91d60 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c91da0 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91df8 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c8e020 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c8df30 common/ctdb_util.c:321 contains 4 bytes in 1 blocks (ref 0) 0x8c8e988 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/events.d contains 68 bytes in 1 blocks (ref 0) 0x8c8fba8 lo contains 3 bytes in 1 blocks (ref 0) 0x8c8ea28 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/test.db/persistent contains 82 bytes in 1 blocks (ref 0) 0x8c8fb20 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/test.db contains 71 bytes in 1 blocks (ref 0) 0x8c8faa0 struct ctdb_node * contains 471 bytes in 18 blocks (ref 0) 0x8c8f3c0 struct ctdb_node contains 165 bytes in 6 blocks (ref 0) 0x8c8f930 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c901e0 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c91208 struct ctdb_tcp_node contains 16 bytes in 1 blocks (ref 0) 0x8c90198 127.0.0.3:4379 contains 15 bytes in 1 blocks (ref 0) 0x8c8f9d8 127.0.0.3 contains 10 bytes in 1 blocks (ref 0) 0x8c8f998 struct ctdb_node contains 165 bytes in 6 blocks (ref 0) 0x8c8f400 struct ctdb_queue contains 72 bytes in 2 blocks (ref 0) 0x8c90140 struct fd_event contains 36 bytes in 1 blocks (ref 0) 0x8c90d40 struct ctdb_tcp_node contains 16 bytes in 1 blocks (ref 0) 0x8c900f8 127.0.0.2:4379 contains 15 bytes in 1 blocks (ref 0) 0x8c8f8e8 127.0.0.2 contains 10 bytes in 1 blocks (ref 0) 0x8c8f468 struct ctdb_node contains 129 bytes in 5 blocks (ref 0) 0x8c8ea60 struct ctdb_queue contains 36 bytes in 1 blocks (ref 0) 0x8c8e0c0 struct ctdb_tcp_node contains 16 bytes in 1 blocks (ref 0) 0x8c8e078 127.0.0.1:4379 contains 15 bytes in 1 blocks (ref 0) 0x8c8f378 127.0.0.1 contains 10 bytes in 1 blocks (ref 0) 0x8c8ebd0 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/nodes.txt contains 73 bytes in 1 blocks (ref 0) 0x8c8e908 tcp contains 4 bytes in 1 blocks (ref 0) 0x8c8e8d0 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/rec.lock contains 72 bytes in 1 blocks (ref 0) 0x8c8e850 struct idr_context contains 2056 bytes in 16 blocks (ref 0) 0x8c8e808 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c91f60 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c91c48 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90c80 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90bc0 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90b00 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90a40 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90980 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c908c0 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90800 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90740 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90680 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c905c0 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90500 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90440 struct idr_layer contains 136 bytes in 1 blocks (ref 0) 0x8c90380 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/daemons.log contains 75 bytes in 1 blocks (ref 0) 0x8c8e770 struct ctdb_log_state contains 1040 bytes in 1 blocks (ref 0) 0x8c8e328 trbt_tree_t contains 8 bytes in 1 blocks (ref 0) 0x8c8e2e8 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 contains 70 bytes in 1 blocks (ref 0) 0x8c8e268 /tmp/ctdb.socket contains 17 bytes in 1 blocks (ref 0) 0x8c8dcf8 struct idr_context contains 16 bytes in 1 blocks (ref 0) 0x8c8dcb0 struct std_event_context contains 28 bytes in 1 blocks (ref 0) 0x8c8da28 autofree_context contains 2142 bytes in 10 blocks (ref 0) 0x8c8d8d0 struct sig_state contains 1976 bytes in 2 blocks (ref 1) 0x8c91380 struct sigaction contains 140 bytes in 1 blocks (ref 0) 0x8c91b30 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8de50 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8de08 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8ddc0 /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/var/sock.0 contains 70 bytes in 1 blocks (ref 0) 0x8c8dd40 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8d998 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8d950 struct event_ops_list contains 16 bytes in 1 blocks (ref 0) 0x8c8d908 There are 115 lines of output Output lines look OK Checking output using '-n all'... There are 399 lines of output Output lines look OK OK: there looks to be output for all 3 nodes Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/25_dumpmemory.sh ========================================================================== TEST PASSED: tests/simple/25_dumpmemory.sh (duration: 2s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/26_ctdb_config_check_error_on_unreachable_ctdb.sh (23:49:56) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY There are 3 nodes. Shutting down node 1... Waiting until node 1 has status "disconnected"... <30|.| OK Output of "! ctdb ip -n 1": 2009/07/16 23:49:58.021874 [31488]: Node 1 is DISCONNECTED OK: "ctdb ip" fails with "disconnected node" Output of "! ctdb disable -n 1": 2009/07/16 23:49:58.048744 [31505]: Node 1 is DISCONNECTED OK: "ctdb disable" fails with "disconnected node" Output of "! ctdb enable -n 1": 2009/07/16 23:49:58.076018 [31520]: Node 1 is DISCONNECTED OK: "ctdb enable" fails with "disconnected node" Output of "! ctdb ban 0 -n 1": 2009/07/16 23:49:58.101394 [31531]: Node 1 is DISCONNECTED OK: "ctdb ban 0" fails with "disconnected node" Output of "! ctdb unban -n 1": 2009/07/16 23:49:58.126218 [31542]: Node 1 is DISCONNECTED OK: "ctdb unban" fails with "disconnected node" Output of "! ctdb listvars -n 1": 2009/07/16 23:49:58.151339 [31553]: Node 1 is DISCONNECTED OK: "ctdb listvars" fails with "disconnected node" That all looks OK. Restarting cluster... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/26_ctdb_config_check_error_on_unreachable_ctdb.sh ========================================================================== TEST PASSED: tests/simple/26_ctdb_config_check_error_on_unreachable_ctdb.sh (duration: 27s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/31_ctdb_disable.sh (23:50:23) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting list of public IPs... Selected node 2 with IPs: 192.0.2.6 192.0.2.3 Disabling node 2 Waiting until node 2 has status "disabled"... <30|| OK Waiting for IPs to fail over... <60|..| OK All IPs moved. Expect a restart... Cluster is UNHEALTHY Restarting ctdb on all nodes to get back into known state... Attempting to politely shutdown daemons... Sleeping for a while... =1|.| Killing remaining daemons... Starting 3 ctdb daemons... Node 0 will have no public IPs. Waiting for cluster to become healthy... <120|..........| OK Setting RerecoveryTimeout to 1 Forcing a recovery... ctdb is ready ALL OK: tests/simple/31_ctdb_disable.sh ========================================================================== TEST PASSED: tests/simple/31_ctdb_disable.sh (duration: 22s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/32_ctdb_enable.sh (23:50:45) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Getting list of public IPs... Selected node 2 with IPs: 192.0.2.6 192.0.2.3 Disabling node 2 Waiting until node 2 has status "disabled"... <30|| OK Waiting for IPs to fail over... <60|..| OK All IPs moved. Reenabling node 2 Waiting until node 2 has status "enabled"... <30|| OK Waiting for IPs to fail over... <60|.| OK All IPs moved. All done! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/32_ctdb_enable.sh ========================================================================== TEST PASSED: tests/simple/32_ctdb_enable.sh (duration: 5s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/41_ctdb_ban.sh (23:50:50) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Finding out which node is the recovery master... Output of "ctdb recmaster": 0 Getting list of public IPs... Selected node 2 with IPs: 192.0.2.6 192.0.2.3 Banning node 2 for 15 seconds Waiting until node 2 has status "banned"... <30|| OK Waiting for IPs to fail over... <60|| OK All IPs moved. Sleeping until ban expires... =15|...............| Waiting until node 2 has status "unbanned"... <30|| OK Waiting for IPs to fail over... <60|| OK All IPs moved. Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/41_ctdb_ban.sh ========================================================================== TEST PASSED: tests/simple/41_ctdb_ban.sh (duration: 19s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/42_ctdb_unban.sh (23:51:09) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Finding out which node is the recovery master... Output of "ctdb recmaster": 0 Getting list of public IPs... Selected node 2 with IPs: 192.0.2.6 192.0.2.3 Banning node 2 for 60 seconds Waiting until node 2 has status "banned"... <30|| OK Waiting for IPs to fail over... <60|| OK All IPs moved. Unbanning node 2 Waiting until node 2 has status "unbanned"... <30|| OK Waiting for IPs to fail over... <60|| OK All IPs moved. Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/42_ctdb_unban.sh ========================================================================== TEST PASSED: tests/simple/42_ctdb_unban.sh (duration: 9s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/51_ctdb_bench.sh (23:51:18) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Running ctdb_bench on all 3 nodes. Output of "/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_bench -n 3": Waiting for cluster Ring: 0.00 msgs/sec (+ve=0 -ve=0) Ring: 0.00 msgs/sec (+ve=0 -ve=0) Ring: 0.00 msgs/sec (+ve=0 -ve=0) Ring: 5971.95 msgs/sec (+ve=4999 -ve=5001) Ring: 8509.81 msgs/sec (+ve=9999 -ve=10001) Ring: 9952.05 msgs/sec (+ve=14998 -ve=15002) Ring: 10871.64 msgs/sec (+ve=19995 -ve=20005) Ring: 11527.99 msgs/sec (+ve=24992 -ve=25008) Ring: 12044.95 msgs/sec (+ve=29990 -ve=30010) Ring: 12418.87 msgs/sec (+ve=34990 -ve=35010) Ring: 12675.93 msgs/sec (+ve=39991 -ve=40009) Ring: 12935.65 msgs/sec (+ve=44988 -ve=45012) Ring: 13130.76 msgs/sec (+ve=49992 -ve=50008) Ring: 13300.74 msgs/sec (+ve=54995 -ve=55005) Ring: 13436.04 msgs/sec (+ve=59996 -ve=60004) Ring: 13549.88 msgs/sec (+ve=64998 -ve=65002) Waiting for cluster Ring: 13618.17 msgs/sec (+ve=68088 -ve=68094) Ring: 13617.37 msgs/sec (+ve=68088 -ve=68094) Waiting for cluster Ring: 13617.45 msgs/sec (+ve=68088 -ve=68095) There are 6 lines of output Output lines look OK OK: 13617.45 msgs/sec >= 10 msgs/sec OK: +ive (68088) > 0 OK: -ive (68095) > 0 OK: percentage difference between +ive and -ive (0%) <= 1% Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/51_ctdb_bench.sh ========================================================================== TEST PASSED: tests/simple/51_ctdb_bench.sh (duration: 12s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/52_ctdb_fetch.sh (23:51:30) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Running ctdb_fetch on all 3 nodes. Output of "/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_fetch -n 3": Waiting for cluster Fetch: 0.00 msgs/sec Fetch: 1804.89 msgs/sec Fetch: 1851.11 msgs/sec Fetch: 1865.61 msgs/sec Fetch: 1865.02 msgs/sec Fetch: 1860.41 msgs/sec Fetch: 1857.60 msgs/sec Fetch: 1857.74 msgs/sec Fetch: 1845.81 msgs/sec Fetch: 1841.04 msgs/sec Fetch: 1831.60 msgs/sec Fetch: 1820.59 msgs/sec Fetch: 1809.41 msgs/sec Fetch: 1801.57 msgs/sec Fetch: 1796.97 msgs/sec Fetch: 1789.97 msgs/sec Fetch: 1784.39 msgs/sec Fetch: 1782.74 msgs/sec Fetch: 1746.06 msgs/sec Fetch: 1743.72 msgs/sec Fetch: 1742.51 msgs/sec Fetch: 1743.05 msgs/sec Fetch: 1741.09 msgs/sec Fetch: 1741.58 msgs/sec Fetch: 1739.69 msgs/sec Fetch: 1736.97 msgs/sec Fetch: 1735.85 msgs/sec Fetch: 1734.78 msgs/sec Fetch: 1734.66 msgs/sec Fetch: 1737.24 msgs/sec Fetch: 1740.81 msgs/sec Fetch: 1741.24 msgs/sec Fetch: 1740.30 msgs/sec Fetch: 1741.42 msgs/sec Fetch: 1743.52 msgs/sec Fetch: 1745.45 msgs/sec Fetch: 1747.21 msgs/sec Fetch: 1749.17 msgs/sec Fetch: 1750.93 msgs/sec Fetch: 1752.16 msgs/sec Fetch: 1753.33 msgs/sec Fetch: 1754.65 msgs/sec Fetch: 1756.30 msgs/sec Fetch: 1757.15 msgs/sec Fetch: 1758.04 msgs/sec Fetch: 1759.67 msgs/sec Fetch: 1760.47 msgs/sec Fetch: 1761.25 msgs/sec Fetch: 1762.09 msgs/sec Fetch: 1763.57 msgs/sec Fetch: 1764.07 msgs/sec Fetch: 1764.89 msgs/sec Fetch: 1766.23 msgs/sec Fetch: 1766.88 msgs/sec Fetch: 1767.23 msgs/sec Fetch: 1767.83 msgs/sec Fetch: 1768.69 msgs/sec Fetch: 1769.71 msgs/sec Fetch: 1770.36 msgs/sec Fetch: 1771.79 msgs/sec Fetch: 1773.05 msgs/sec Fetch: 1773.50 msgs/sec Fetch: 1774.59 msgs/sec Fetch: 1775.83 msgs/sec Fetch: 1777.36 msgs/sec Fetch: 1778.41 msgs/sec Fetch: 1774.58 msgs/sec Fetch: 1776.34 msgs/sec Fetch: 1777.55 msgs/sec Fetch: 1778.78 msgs/sec Fetch: 1779.89 msgs/sec Fetch: 1781.29 msgs/sec Fetch: 1782.15 msgs/sec Fetch: 1782.57 msgs/sec Fetch: 1783.47 msgs/sec Fetch: 1783.81 msgs/sec Fetch: 1784.00 msgs/sec Fetch: 1784.28 msgs/sec Fetch: 1784.68 msgs/sec Fetch: 1785.26 msgs/sec Fetch: 1785.62 msgs/sec Fetch: 1786.30 msgs/sec Fetch: 1786.65 msgs/sec Fetch: 1787.01 msgs/sec Fetch: 1787.06 msgs/sec Fetch: 1787.22 msgs/sec Fetch: 1787.66 msgs/sec Fetch: 1787.81 msgs/sec Fetch: 1788.00 msgs/sec Fetch: 1788.61 msgs/sec Fetch: 1788.78 msgs/sec Fetch: 1789.03 msgs/sec Fetch: 1789.25 msgs/sec Fetch: 1789.66 msgs/sec Fetch: 1789.73 msgs/sec Fetch: 1789.85 msgs/sec Fetch: 1790.10 msgs/sec Fetch: 1790.21 msgs/sec Fetch: 1790.28 msgs/sec Fetch: 1790.52 msgs/sec Fetch: 1791.00 msgs/sec Fetch: 1791.27 msgs/sec Fetch: 1791.48 msgs/sec Fetch: 1791.91 msgs/sec Fetch: 1792.10 msgs/sec Fetch: 1792.31 msgs/sec Fetch: 1792.54 msgs/sec Fetch: 1792.31 msgs/sec Fetch: 1792.84 msgs/sec Fetch: 1793.15 msgs/sec Fetch: 1793.67 msgs/sec Fetch: 1794.07 msgs/sec Fetch: 1794.43 msgs/sec Fetch: 1794.76 msgs/sec Fetch: 1795.10 msgs/sec Fetch: 1795.84 msgs/sec Fetch: 1796.31 msgs/sec Fetch: 1796.52 msgs/sec Fetch: 1797.28 msgs/sec Fetch: 1797.56 msgs/sec Fetch: 1797.82 msgs/sec Fetch: 1798.03 msgs/sec Fetch: 1798.40 msgs/sec Fetch: 1798.84 msgs/sec Fetch: 1799.07 msgs/sec Fetch: 1799.71 msgs/sec Fetch: 1800.02 msgs/sec Fetch: 1800.44 msgs/sec Fetch: 1800.78 msgs/sec Fetch: 1801.06 msgs/sec Fetch: 1801.58 msgs/sec Fetch: 1801.80 msgs/sec Fetch: 1801.99 msgs/sec Fetch: 1802.50 msgs/sec Fetch: 1802.75 msgs/sec Fetch: 1803.02 msgs/sec Fetch: 1803.23 msgs/sec Fetch: 1803.99 msgs/sec Fetch: 1804.14 msgs/sec Fetch: 1801.95 msgs/sec Fetch: 1802.27 msgs/sec Fetch: 1802.61 msgs/sec Fetch: 1803.04 msgs/sec Fetch: 1803.48 msgs/sec Fetch: 1803.92 msgs/sec Fetch: 1804.43 msgs/sec Fetch: 1804.71 msgs/sec Fetch: 1805.10 msgs/sec Fetch: 1805.51 msgs/sec Fetch: 1803.67 msgs/sec Fetch: 1804.00 msgs/sec Fetch: 1804.26 msgs/sec Fetch: 1804.68 msgs/sec Fetch: 1804.94 msgs/sec Fetch: 1805.24 msgs/sec Fetch: 1805.68 msgs/sec Fetch: 1805.91 msgs/sec Fetch: 1806.22 msgs/sec Fetch: 1806.51 msgs/sec Fetch: 1806.84 msgs/sec Fetch: 1807.10 msgs/sec Fetch: 1807.30 msgs/sec Fetch: 1807.76 msgs/sec Fetch: 1808.10 msgs/sec Fetch: 1808.05 msgs/sec Fetch: 1808.30 msgs/sec Fetch: 1808.52 msgs/sec Fetch: 1808.88 msgs/sec Fetch: 1809.14 msgs/sec Fetch: 1809.39 msgs/sec Fetch: 1809.87 msgs/sec Fetch: 1810.12 msgs/sec Fetch: 1810.30 msgs/sec Fetch: 1810.54 msgs/sec Fetch: 1810.95 msgs/sec Fetch: 1811.13 msgs/sec Fetch: 1811.33 msgs/sec Fetch: 1811.66 msgs/sec Fetch: 1811.84 msgs/sec Fetch: 1812.03 msgs/sec Fetch: 1812.19 msgs/sec Fetch: 1812.35 msgs/sec Fetch: 1812.46 msgs/sec Fetching final record DATA: Test data msg_count=18124 on node 1 msg_count=18124 on node 2 msg_count=18125 on node 0 Waiting for cluster Fetch: 453.14 msgs/sec Fetching final record DATA: Test data msg_count=18124 on node 1 msg_count=18124 on node 2 msg_count=18125 on node 0 msg_count=18125 on node 1 msg_count=18125 on node 2 Waiting for cluster Fetch: 453.14 msgs/sec Fetching final record DATA: è¬ о  There are 25 lines of output BAD: unexpected lines in output: Fetch: 0.00 msgs/sec^MFetch: 1804.89 msgs/sec^MFetch: 1851.11 msgs/sec^MFetch: 1865.61 msgs/sec^MFetch: 1865.02 msgs/sec^MFetch: 1860.41 msgs/sec^MFetch: 1857.60 msgs/sec^MFetch: 1857.74 msgs/sec^MFetch: 1845.81 msgs/sec^MFetch: 1841.04 msgs/sec^MFetch: 1831.60 msgs/sec^MFetch: 1820.59 msgs/sec^MFetch: 1809.41 msgs/sec^MFetch: 1801.57 msgs/sec^MFetch: 1796.97 msgs/sec^MFetch: 1789.97 msgs/sec^MFetch: 1784.39 msgs/sec^MFetch: 1782.74 msgs/sec^MFetch: 1746.06 msgs/sec^MFetch: 1743.72 msgs/sec^MFetch: 1742.51 msgs/sec^MFetch: 1743.05 msgs/sec^MFetch: 1741.09 msgs/sec^MFetch: 1741.58 msgs/sec^MFetch: 1739.69 msgs/sec^MFetch: 1736.97 msgs/sec^MFetch: 1735.85 msgs/sec^MFetch: 1734.78 msgs/sec^MFetch: 1734.66 msgs/sec^MFetch: 1737.24 msgs/sec^MFetch: 1740.81 msgs/sec^MFetch: 1741.24 msgs/sec^MFetch: 1740.30 msgs/sec^MFetch: 1741.42 msgs/sec^MFetch: 1743.52 msgs/sec^MFetch: 1745.45 msgs/sec^MFetch: 1747.21 msgs/sec^MFetch: 1749.17 msgs/sec^MFetch: 1750.93 msgs/sec^MFetch: 1752.16 msgs/sec^MFetch: 1753.33 msgs/sec^MFetch: 1754.65 msgs/sec^MFetch: 1756.30 msgs/sec^MFetch: 1757.15 msgs/sec^MFetch: 1758.04 msgs/sec^MFetch: 1759.67 msgs/sec^MFetch: 1760.47 msgs/sec^MFetch: 1761.25 msgs/sec^MFetch: 1762.09 msgs/sec^MFetch: 1763.57 msgs/sec^MFetch: 1764.07 msgs/sec^MFetch: 1764.89 msgs/sec^MFetch: 1766.23 msgs/sec^MFetch: 1766.88 msgs/sec^MFetch: 1767.23 msgs/sec^MFetch: 1767.83 msgs/sec^MFetch: 1768.69 msgs/sec^MFetch: 1769.71 msgs/sec^MFetch: 1770.36 msgs/sec^MFetch: 1771.79 msgs/sec^MFetch: 1773.05 msgs/sec^MFetch: 1773.50 msgs/sec^MFetch: 1774.59 msgs/sec^MFetch: 1775.83 msgs/sec^MFetch: 1777.36 msgs/sec^MFetch: 1778.41 msgs/sec^MFetch: 1774.58 msgs/sec^MFetch: 1776.34 msgs/sec^MFetch: 1777.55 msgs/sec^MFetch: 1778.78 msgs/sec^MFetch: 1779.89 msgs/sec^MFetch: 1781.29 msgs/sec^MFetch: 1782.15 msgs/sec^MFetch: 1782.57 msgs/sec^MFetch: 1783.47 msgs/sec^MFetch: 1783.81 msgs/sec^MFetch: 1784.00 msgs/sec^MFetch: 1784.28 msgs/sec^MFetch: 1784.68 msgs/sec^MFetch: 1785.26 msgs/sec^MFetch: 1785.62 msgs/sec^MFetch: 1786.30 msgs/sec^MFetch: 1786.65 msgs/sec^MFetch: 1787.01 msgs/sec^MFetch: 1787.06 msgs/sec^MFetch: 1787.22 msgs/sec^MFetch: 1787.66 msgs/sec^MFetch: 1787.81 msgs/sec^MFetch: 1788.00 msgs/sec^MFetch: 1788.61 msgs/sec^MFetch: 1788.78 msgs/sec^MFetch: 1789.03 msgs/sec^MFetch: 1789.25 msgs/sec^MFetch: 1789.66 msgs/sec^MFetch: 1789.73 msgs/sec^MFetch: 1789.85 msgs/sec^MFetch: 1790.10 msgs/sec^MFetch: 1790.21 msgs/sec^MFetch: 1790.28 msgs/sec^MFetch: 1790.52 msgs/sec^MFetch: 1791.00 msgs/sec^MFetch: 1791.27 msgs/sec^MFetch: 1791.48 msgs/sec^MFetch: 1791.91 msgs/sec^MFetch: 1792.10 msgs/sec^MFetch: 1792.31 msgs/sec^MFetch: 1792.54 msgs/sec^MFetch: 1792.31 msgs/sec^MFetch: 1792.84 msgs/sec^MFetch: 1793.15 msgs/sec^MFetch: 1793.67 msgs/sec^MFetch: 1794.07 msgs/sec^MFetch: 1794.43 msgs/sec^MFetch: 1794.76 msgs/sec^MFetch: 1795.10 msgs/sec^MFetch: 1795.84 msgs/sec^MFetch: 1796.31 msgs/sec^MFetch: 1796.52 msgs/sec^MFetch: 1797.28 msgs/sec^MFetch: 1797.56 msgs/sec^MFetch: 1797.82 msgs/sec^MFetch: 1798.03 msgs/sec^MFetch: 1798.40 msgs/sec^MFetch: 1798.84 msgs/sec^MFetch: 1799.07 msgs/sec^MFetch: 1799.71 msgs/sec^MFetch: 1800.02 msgs/sec^MFetch: 1800.44 msgs/sec^MFetch: 1800.78 msgs/sec^MFetch: 1801.06 msgs/sec^MFetch: 1801.58 msgs/sec^MFetch: 1801.80 msgs/sec^MFetch: 1801.99 msgs/sec^MFetch: 1802.50 msgs/sec^MFetch: 1802.75 msgs/sec^MFetch: 1803.02 msgs/sec^MFetch: 1803.23 msgs/sec^MFetch: 1803.99 msgs/sec^MFetch: 1804.14 msgs/sec^MFetch: 1801.95 msgs/sec^MFetch: 1802.27 msgs/sec^MFetch: 1802.61 msgs/sec^MFetch: 1803.04 msgs/sec^MFetch: 1803.48 msgs/sec^MFetch: 1803.92 msgs/sec^MFetch: 1804.43 msgs/sec^MFetch: 1804.71 msgs/sec^MFetch: 1805.10 msgs/sec^MFetch: 1805.51 msgs/sec^MFetch: 1803.67 msgs/sec^MFetch: 1804.00 msgs/sec^MFetch: 1804.26 msgs/sec^MFetch: 1804.68 msgs/sec^MFetch: 1804.94 msgs/sec^MFetch: 1805.24 msgs/sec^MFetch: 1805.68 msgs/sec^MFetch: 1805.91 msgs/sec^MFetch: 1806.22 msgs/sec^MFetch: 1806.51 msgs/sec^MFetch: 1806.84 msgs/sec^MFetch: 1807.10 msgs/sec^MFetch: 1807.30 msgs/sec^MFetch: 1807.76 msgs/sec^MFetch: 1808.10 msgs/sec^MFetch: 1808.05 msgs/sec^MFetch: 1808.30 msgs/sec^MFetch: 1808.52 msgs/sec^MFetch: 1808.88 msgs/sec^MFetch: 1809.14 msgs/sec^MFetch: 1809.39 msgs/sec^MFetch: 1809.87 msgs/sec^MFetch: 1810.12 msgs/sec^MFetch: 1810.30 msgs/sec^MFetch: 1810.54 msgs/sec^MFetch: 1810.95 msgs/sec^MFetch: 1811.13 msgs/sec^MFetch: 1811.33 msgs/sec^MFetch: 1811.66 msgs/sec^MFetch: 1811.84 msgs/sec^MFetch: 1812.03 msgs/sec^MFetch: 1812.19 msgs/sec^MFetch: 1812.35 msgs/sec^MFetch: 1812.46 msgs/sec$ Fetching final record$ DATA:$ Test data$ msg_count=18124 on node 1$ msg_count=18124 on node 2$ msg_count=18125 on node 0$ $ Fetch: 453.14 msgs/sec$ Fetching final record$ DATA:$ Test data$ msg_count=18124 on node 1$ msg_count=18124 on node 2$ msg_count=18125 on node 0$ msg_count=18125 on node 1$ msg_count=18125 on node 2$ $ Fetch: 453.14 msgs/sec$ Fetching final record$ DATA:$ M-hM-,^[^IM-PM->^[^I^[^Q$ Cluster is HEALTHY Forcing a recovery... ========================================================================== TEST FAILED: tests/simple/52_ctdb_fetch.sh (status 1) (duration: 41s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/53_ctdb_transaction.sh (23:52:11) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Running ctdb_transaction on all 3 nodes. Output of "/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_transaction --timelimit=30 & /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_transaction --timelimit=30": Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Waiting for cluster Starting test on node 1. running for 30 seconds Starting test on node 1. running for 30 seconds Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Waiting for cluster Starting test on node 2. running for 30 seconds Starting test on node 2. running for 30 seconds [2060] Counters: 4 1 3 [2063] Counters: 6 1 5 [2060] Counters: 8 4 9 [2063] Counters: 9 4 10 [2060] Counters: 12 7 15 [2063] Counters: 9 4 10 [2060] Counters: 16 10 19 [2063] Counters: 15 10 18 [2060] Counters: 19 12 21 [2063] Counters: 20 12 22 [2060] Counters: 23 15 25 [2063] Counters: 21 13 24 [2060] Counters: 27 19 30 [2063] Counters: 26 19 29 [2060] Counters: 27 19 30 [2063] Counters: 29 23 32 [2060] Counters: 30 26 35 [2063] Counters: 32 28 37 [2060] Counters: 34 30 39 [2063] Counters: 33 29 38 [2060] Counters: 38 35 43 [2063] Counters: 37 34 42 [2060] Counters: 41 38 47 [2063] Counters: 40 37 47 [2060] Counters: 42 42 50 [2063] Counters: 43 43 51 [2060] Counters: 47 46 54 [2063] Counters: 46 45 54 [2060] Counters: 50 48 57 [2063] Counters: 49 48 56 [2060] Counters: 54 52 61 [2063] Counters: 52 51 60 [2060] Counters: 57 55 64 [2063] Counters: 55 53 61 [2060] Counters: 59 56 66 [2063] Counters: 61 58 68 [2063] Counters: 61 58 68 [2060] Counters: 65 61 71 [2060] Counters: 65 61 71 [2063] Counters: 68 66 74 [2060] Counters: 72 68 78 [2063] Counters: 71 68 77 [2060] Counters: 73 68 79 [2063] Counters: 71 68 77 [2060] Counters: 75 73 84 [2063] Counters: 76 74 86 [2060] Counters: 78 78 90 [2063] Counters: 76 74 86 [2060] Counters: 81 81 93 [2063] Counters: 82 82 94 [2060] Counters: 85 84 97 [2063] Counters: 84 84 96 [2060] Counters: 89 88 101 [2063] Counters: 90 88 102 [2060] Counters: 92 90 103 [2063] Counters: 94 91 105 [2060] Counters: 92 90 103 [2063] Counters: 96 95 108 [2060] Counters: 98 99 110 SUCCESS! SUCCESS! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/53_ctdb_transaction.sh ========================================================================== TEST PASSED: tests/simple/53_ctdb_transaction.sh (duration: 32s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/61_ctdb_persistent_safe.sh (23:52:43) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Running ctdb_persistent on all 3 nodes. Output of "/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_persistent --timelimit=30 & /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_persistent --timelimit=30": Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Starting test on node 1. running for 30 seconds Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Starting test on node 1. running for 30 seconds Waiting for cluster Starting test on node 2. running for 30 seconds Waiting for cluster Starting test on node 2. running for 30 seconds [5266] Counters: 8 5 5 [5270] Counters: 6 3 3 [5266] Counters: 8 5 5 [5270] Counters: 14 14 14 [5270] Counters: 18 19 19 [5266] Counters: 20 21 20 [5266] Counters: 28 27 26 [5270] Counters: 29 28 26 [5266] Counters: 35 32 32 [5270] Counters: 36 33 32 [5266] Counters: 43 40 39 [5270] Counters: 41 38 36 [5266] Counters: 47 42 43 [5270] Counters: 50 44 44 [5266] Counters: 55 51 50 [5270] Counters: 53 49 49 [5266] Counters: 58 57 57 [5270] Counters: 59 57 58 [5266] Counters: 64 66 64 [5270] Counters: 61 60 61 [5266] Counters: 71 72 69 [5270] Counters: 69 71 68 [5266] Counters: 75 76 72 [5270] Counters: 78 77 75 [5266] Counters: 85 82 81 [5270] Counters: 82 79 78 [5266] Counters: 90 89 86 [5270] Counters: 91 90 87 [5266] Counters: 95 94 92 [5270] Counters: 96 95 94 [5266] Counters: 104 99 101 [5270] Counters: 102 98 100 [5266] Counters: 110 107 106 [5270] Counters: 105 100 102 [5266] Counters: 115 115 111 [5270] Counters: 113 111 109 [5266] Counters: 120 122 117 [5270] Counters: 121 123 117 [5266] Counters: 127 128 123 [5270] Counters: 126 127 121 [5266] Counters: 134 136 130 [5270] Counters: 133 135 129 [5270] Counters: 141 141 135 [5266] Counters: 142 142 135 [5270] Counters: 143 143 136 [5266] Counters: 145 144 138 [5270] Counters: 151 155 147 [5266] Counters: 152 156 148 [5270] Counters: 158 158 152 [5266] Counters: 161 159 154 [5270] Counters: 165 164 158 [5266] Counters: 171 167 160 [5270] Counters: 175 171 164 [5266] Counters: 179 175 166 [5270] Counters: 183 179 170 [5266] Counters: 187 180 174 [5270] Counters: 196 187 181 [5266] Counters: 194 186 178 [5270] Counters: 199 190 182 [5266] Counters: 203 194 185 SUCCESS! SUCCESS! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/61_ctdb_persistent_safe.sh ========================================================================== TEST PASSED: tests/simple/61_ctdb_persistent_safe.sh (duration: 31s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/62_ctdb_persistent_unsafe.sh (23:53:14) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Cluster is HEALTHY Running ctdb_persistent --unsafe-writes on all 3 nodes. Output of "/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_persistent --unsafe-writes --timelimit=30 & /var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/scripts/test_wrap ctdb_persistent --unsafe-writes --timelimit=30": Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Starting test on node 1. running for 30 seconds Waiting for cluster Starting test on node 1. running for 30 seconds Waiting for cluster Starting test on node 0. running for 30 seconds Waiting for cluster Waiting for cluster Starting test on node 2. running for 30 seconds Starting test on node 2. running for 30 seconds [11311] Counters: 413 403 383 [11321] Counters: 418 407 392 [11311] Counters: 632 611 592 [11321] Counters: 636 616 598 [11311] Counters: 864 820 830 [11321] Counters: 868 825 835 [11311] Counters: 1079 1051 1056 [11321] Counters: 1086 1055 1059 [11311] Counters: 1296 1279 1283 [11321] Counters: 1299 1280 1286 [11311] Counters: 1511 1492 1515 [11321] Counters: 1514 1494 1517 [11311] Counters: 1708 1708 1729 [11321] Counters: 1713 1712 1731 [11311] Counters: 1936 1908 1947 [11321] Counters: 1941 1912 1952 [11311] Counters: 2144 2145 2177 [11321] Counters: 2149 2149 2180 [11311] Counters: 2370 2351 2406 [11321] Counters: 2372 2353 2408 [11311] Counters: 2589 2582 2614 [11321] Counters: 2592 2586 2619 [11311] Counters: 2816 2798 2834 [11321] Counters: 2813 2797 2833 [11311] Counters: 3029 3011 3047 [11321] Counters: 3033 3014 3051 [11311] Counters: 3249 3227 3259 [11321] Counters: 3250 3227 3260 [11311] Counters: 3465 3452 3467 [11321] Counters: 3467 3455 3470 [11311] Counters: 3688 3670 3701 [11321] Counters: 3694 3673 3705 [11311] Counters: 3912 3898 3922 [11321] Counters: 3914 3900 3924 [11311] Counters: 4124 4115 4149 [11321] Counters: 4125 4116 4150 [11311] Counters: 4336 4325 4375 [11321] Counters: 4341 4331 4381 [11311] Counters: 4541 4548 4610 [11321] Counters: 4544 4549 4614 [11311] Counters: 4747 4767 4826 [11321] Counters: 4751 4772 4828 [11311] Counters: 4974 4987 5052 [11321] Counters: 4979 4989 5057 [11311] Counters: 5178 5202 5276 [11321] Counters: 5171 5197 5270 [11311] Counters: 5387 5420 5498 [11321] Counters: 5391 5423 5499 [11311] Counters: 5600 5644 5718 [11321] Counters: 5599 5642 5716 [11311] Counters: 5823 5871 5935 [11321] Counters: 5827 5875 5939 [11311] Counters: 6052 6099 6159 [11321] Counters: 6055 6103 6161 [11311] Counters: 6270 6322 6381 [11321] Counters: 6273 6324 6383 [11311] Counters: 6490 6552 6607 [11321] Counters: 6487 6548 6603 SUCCESS! SUCCESS! Cluster is HEALTHY Forcing a recovery... ALL OK: tests/simple/62_ctdb_persistent_unsafe.sh ========================================================================== TEST PASSED: tests/simple/62_ctdb_persistent_unsafe.sh (duration: 31s) ========================================================================== --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- Running test tests/simple/99_ctdb_uninstall_eventscript.sh (23:53:45) --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-- removed `/var/tmp/portage/dev-db/ctdb-1.0.73/work/ctdb-1.0.73/tests/events.d/00.ctdb_test_trigger' ALL OK: tests/simple/99_ctdb_uninstall_eventscript.sh ========================================================================== TEST PASSED: tests/simple/99_ctdb_uninstall_eventscript.sh (duration: 0s) ========================================================================== tests/simple/00_ctdb_init.sh PASSED tests/simple/00_ctdb_install_eventscript.sh PASSED tests/simple/00_ctdb_onnode.sh PASSED tests/simple/01_ctdb_version.sh PASSED tests/simple/02_ctdb_listvars.sh PASSED tests/simple/03_ctdb_getvar.sh PASSED tests/simple/04_ctdb_setvar.sh PASSED tests/simple/05_ctdb_listnodes.sh PASSED tests/simple/06_ctdb_getpid.sh PASSED tests/simple/07_ctdb_process_exists.sh PASSED tests/simple/08_ctdb_isnotrecmaster.sh PASSED tests/simple/09_ctdb_ping.sh PASSED tests/simple/11_ctdb_ip.sh PASSED tests/simple/12_ctdb_getdebug.sh PASSED tests/simple/13_ctdb_setdebug.sh PASSED tests/simple/14_ctdb_statistics.sh PASSED tests/simple/15_ctdb_statisticsreset.sh PASSED tests/simple/16_ctdb_config_add_ip.sh PASSED tests/simple/17_ctdb_config_delete_ip.sh PASSED tests/simple/18_ctdb_freeze.sh PASSED tests/simple/19_ctdb_thaw.sh PASSED tests/simple/20_ctdb_getmonmode.sh PASSED tests/simple/21_ctdb_disablemonitor.sh PASSED tests/simple/22_ctdb_enablemonitor.sh PASSED tests/simple/23_ctdb_moveip.sh PASSED tests/simple/24_ctdb_getdbmap.sh PASSED tests/simple/25_dumpmemory.sh PASSED tests/simple/26_ctdb_config_check_error_on_unreachable_ctdb.sh PASSED tests/simple/31_ctdb_disable.sh PASSED tests/simple/32_ctdb_enable.sh PASSED tests/simple/41_ctdb_ban.sh PASSED tests/simple/42_ctdb_unban.sh PASSED tests/simple/51_ctdb_bench.sh PASSED tests/simple/52_ctdb_fetch.sh FAILED tests/simple/53_ctdb_transaction.sh PASSED tests/simple/61_ctdb_persistent_safe.sh PASSED tests/simple/62_ctdb_persistent_unsafe.sh PASSED tests/simple/99_ctdb_uninstall_eventscript.sh PASSED 37/38 tests passed make: *** [test] Error 1 * * ERROR: dev-db/ctdb-1.0.73 failed. * Call stack: * ebuild.sh, line 49: Called src_test * environment, line 2862: Called _eapi0_src_test * ebuild.sh, line 613: Called die * The specific snippet of code: * hasq test $FEATURES && die "Make test failed. See above for details." * The die message: * Make test failed. See above for details. * * If you need support, post the topmost build error, and the call stack if relevant. * A complete build log is located at '/var/log/portage/dev-db/ctdb-1.0.73:20090716-234334.log'. * The ebuild environment file is located at '/var/tmp/portage/dev-db/ctdb-1.0.73/temp/environment'. * >>> Install ctdb-1.0.73 into /var/tmp/portage/dev-db/ctdb-1.0.73/image/ category dev-db make -j14 DESTDIR=/var/tmp/portage/dev-db/ctdb-1.0.73/image/ install ctdb will be compiled with flags: CFLAGS = -g -I./include -Iinclude -Ilib -Ilib/util -I. -I./lib/talloc -I./lib/tdb/include -I./lib/replace -DVARDIR=\"/var/lib\" -DETCDIR=\"/etc\" -DLOGDIR=\"/var/lib/log\" -DUSE_MMAP=1 -O2 -pipe -I./lib/replace -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings LIBS = mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/sbin mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/include mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d mkdir -p /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/doc/ctdb /usr/bin/install -c -m 755 bin/ctdb /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin /usr/bin/install -c -m 755 bin/ctdb_ipmux /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin /usr/bin/install -c -m 755 bin/ctdbd /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/sbin /usr/bin/install -c -m 755 bin/smnotify /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin /usr/bin/install -c -m 644 include/ctdb.h /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/include /usr/bin/install -c -m 644 include/ctdb_private.h /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/include # for samba3 /usr/bin/install -c -m 644 config/functions /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb /usr/bin/install -c -m 755 config/statd-callout /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb /usr/bin/install -c -m 644 config/events.d/README /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/doc/ctdb/README.eventscripts /usr/bin/install -c -m 644 doc/recovery-process.txt /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/doc/ctdb/recovery-process.txt /usr/bin/install -c -m 755 config/events.d/00.ctdb /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/10.interface /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/20.multipathd /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/40.vsftpd /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/41.httpd /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/50.samba /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/60.nfs /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/61.nfstickle /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/70.iscsi /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/90.ipmux /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/91.lvs /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 config/events.d/99.routing /var/tmp/portage/dev-db/ctdb-1.0.73/image//etc/ctdb/events.d /usr/bin/install -c -m 755 tools/ctdb_diagnostics /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin /usr/bin/install -c -m 755 tools/onnode /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/bin if [ -f doc/ctdb.1 ];then /usr/bin/install -c -d /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/man/man1; fi if [ -f doc/ctdb.1 ];then /usr/bin/install -c -m 644 doc/ctdb.1 /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/man/man1; fi if [ -f doc/ctdbd.1 ];then /usr/bin/install -c -m 644 doc/ctdbd.1 /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/man/man1; fi if [ -f doc/onnode.1 ];then /usr/bin/install -c -m 644 doc/onnode.1 /var/tmp/portage/dev-db/ctdb-1.0.73/image//usr/share/man/man1; fi >>> Completed installing ctdb-1.0.73 into /var/tmp/portage/dev-db/ctdb-1.0.73/image/ strip: i686-pc-linux-gnu-strip --strip-unneeded -R .comment usr/sbin/ctdbd usr/bin/smnotify usr/bin/ctdb_ipmux usr/bin/ctdb * Removing /usr/share/doc * Removing /usr/share/man * Removing /usr/share/info