We recently deployed a two node OCFS2 cluster, sitting on top of DRBD. Everything worked great in limited testing, but once we went into production it started to fail. The failure was encountered whenever there was a symlink in a directory on the OCFS2 volume, and it was accessed with anything like ls, stat, etc. The fault produced a buffer overflow in the kernel. Unfortunately, this was all weeks ago at this point, and I failed to keep the error reports, but it's simple enough to reproduce. I was able to find a fix for the issue here: https://oss.oracle.com/pipermail/ocfs2-devel/2012-August/008677.html Reproducible: Always Steps to Reproduce: 1. Create ocfs2 volume and mount it (/mnt/ocfs2vol) 2. Create the symlink: mkdir -p /mnt/ocfs2vol/test touch /mnt/ocfs2vol/test/testfile ln -snf /mnt/ocfs2vol/test/testfile /mnt/ocfs2vol/testlink 3. Access the directory: ls -l /mnt/ocfs2vol/test/ Observe kernel messages Actual Results: Buffer overflow error reported by the kernel. Expected Results: Accessing the directory. I will be putting together a new cluster in the near future and will be able to reproduce this completely and have full output. The fix that I found does work, and has been submitted to Oracle judging by the thread. Where in the kernel commits this is I have no idea. # emerge --info Portage 2.1.12.2 (default/linux/amd64/13.0/no-multilib, gcc-4.6.3, glibc-2.15-r3, 3.8.13-gentoo x86_64) ================================================================= System uname: Linux-3.8.13-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_X5650_@_2.67GHz-with-gentoo-2.2 KiB Mem: 32932752 total, 176144 free KiB Swap: 2097148 total, 2092228 free Timestamp of tree: Tue, 06 Aug 2013 04:15:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5, 3.2.5-r1 dev-util/cmake: 2.8.10.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.11.6, 1.12.6 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.7 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps y" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.cs.uni.edu/ http://mirror.usu.edu/mirrors/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j25" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://192.168.0.157/gentoo-portage" USE="acl amd64 apache2 bash-completion berkdb bzip2 cli corefonts cracklib crypt ctype curl cxx djvu dri fontconfig fortran fpx ftp gcj gd gdbm gif gnutls graphviz gs hash hdri iconv imagemagick ipv6 jbig jpeg jpeg2k lcms lzma mmx modules mudflap ncurses nls nptl nptlonly openexr openmp openssl pam pcntl pcre pdf png posix readline samba sasl session simplexml smtp snmp soap sockets sse sse2 ssh ssl svg tcpd threads tiff truetype unicode vim-syntax webp wmf xml zip zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
If it's of any worth, my ocfs2 mount in /etc/fstab: # grep -P '^/.*ocfs2' /etc/fstab /dev/drbd1 /mnt/storage ocfs2 noatime,nodiratime,_netdev,noauto 0 0 The ocfs2 configuration: # cat /etc/ocfs2/cluster.conf cluster: heartbeat_mode = global node_count = 2 name = cluster* node: number = 1 cluster = cluster* ip_port = 7777 ip_address = 172.16.0.10 name = host1* node: number = 2 cluster = cluster* ip_port = 7777 ip_address = 172.16.0.11 name = host2* heartbeat: cluster = cluster* region = 14DF63D68F504B188E4370E0C31523C3 # cat /etc/conf.d/ocfs2 # Copyright 1999-2011 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/sys-fs/ocfs2-tools/files/ocfs2.confd,v 1.1 2011/08/31 17:15:18 alexxy Exp $ # Put your cluster names here, separated by space, ie. # OCFS2_CLUSTER="cluster1 admincluster cluster2" OCFS2_CLUSTER="cluster*" # Some heartbeat tweaks to prevent self-fencing quite so much during heavy load. # http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html # How long to wait before a node is considered dead from lack of network activity. OCFS2_IDLE_TIMEOUT_MS="30000" # How often we should attempt to send heartbeats. OCFS2_KEEPALIVE_DELAY_MS="2000" OCFS2_RECONNECT_DELAY_MS="2000" # How often we should attempt to send heartbeats. # How many interations before a node is considered dead from lack of IO activity. # (dead_threshold - 1) * 2s OCFS2_DEAD_THRESHOLD="61" # Default: "-fy" (force check on double-unmounted boot). I beleave this is best. # Native way is "-y", but once I have twice reboot/powerOFF with bad result... # Affected only fstab. OCFS2_FSCK="-fy" # http://oss.oracle.com/projects/ocfs2-tools/news/article_8.html # starting from 1.4.3, fsck.ocfs2 "aggressively cache the metadata blocks" # this is useful on lots of free memory, but I have heavy slowdown on x86_64 # with 1G of RAM. # "yes" will "swapoff -a" & "swapon -a" around fsck to avoid caching over swap OCFS2_FSCK_SWAPOFF="yes" # Signal to kill processes on stop/umount on busy device (empty to not kill) #OCFS2_UMOUNT_KILL="KILL" # "yes" to force stop #OCFS2_FORCE_STOP="no" * names have been changed for reporting
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=30b9c9e6ba289ba3bb67cc292efcc4122ea37ae5 # git tag --contains 30b9c9e6ba289ba3bb67cc292efcc4122ea37ae5 v3.10 v3.10-rc1 v3.10-rc2 v3.10-rc3 v3.10-rc4 v3.10-rc5 v3.10-rc6 v3.10-rc7 v3.10.1 v3.10.2 v3.10.3 v3.10.4 v3.10.5 v3.10.6 v3.11-rc1 v3.11-rc2 v3.11-rc3 v3.11-rc4 v3.11-rc5 v3.9 v3.9-rc1 v3.9-rc2 v3.9-rc3 v3.9-rc4 v3.9-rc5 v3.9-rc6 v3.9-rc7 v3.9-rc8 v3.9.1 v3.9.10 v3.9.11 v3.9.2 v3.9.3 v3.9.4 v3.9.5 v3.9.6 v3.9.7 v3.9.8 v3.9.9 Fixed since v3.9-rc1.
