Bug 364903 – =net-fs/nfs-utils-1.2.3-r1 rpc.mountd memory leak

• Product: Gentoo Linux
• Status: RESOLVED TEST-REQUEST

[Description Spooky Ghost 2011-04-26 09:32:23 UTC]

There seems to be a memory leak in rpc.mountd such that after 4 weeks of uptime the process is using ~2.5Gb memory.  If I restart the machine or nfs services then memory usage is acceptable but will grow over time.  I have around 10 lines in /etc/exports including entries for distfiles, ccache and /home.  These are all automounted to around 10 machines (physical and xen guests).  Kernel on the server is xen pv_ops 2.6.32 stable tree (2.6.32.35-05163-gdf3a556).

Reproducible: Always

Steps to Reproduce:
1. Boot machine
2. Wait
3. Check memory usage of rpc.mountd
Actual Results:  
Memory allocation for rpc.mountd grows over time

Expected Results:  
Memory allocation for rpc.mountd is reasoanbly consistent over time

# cat /proc/11625/status
Name:   rpc.mountd
State:  S (sleeping)
Tgid:   11625
Pid:    11625
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 64
Groups:
VmPeak:  2659972 kB
VmSize:  2659344 kB
VmLck:         0 kB
VmHWM:    918764 kB
VmRSS:    448440 kB
VmData:  2596316 kB
VmStk:        88 kB
VmExe:        96 kB
VmLib:      7156 kB
VmPTE:      5208 kB
Threads:        1
SigQ:   3/13611
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000011000
SigCgt: 0000000180004a03
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
voluntary_ctxt_switches:        620552
nonvoluntary_ctxt_switches:     729699



$ emerge --info
Portage 2.1.9.42 (default/linux/amd64/10.0/no-multilib, gcc-4.4.5, glibc-2.11.3-r0, 2.6.32.35-05163-gdf3a556 x86_64)
=================================================================
System uname: Linux-2.6.32.35-05163-gdf3a556-x86_64-Six-Core_AMD_Opteron-tm-_Processor_2423_HE-with-gentoo-1.12.14
Timestamp of tree: Tue, 26 Apr 2011 07:45:01 +0000
ccache version 3.1.4 [enabled]
app-shells/bash:     4.1_p9
dev-java/java-config: 1.3.7-r1, 2.1.11-r3
dev-lang/python:     2.4.6, 2.5.4-r4, 2.6.6-r2, 3.1.3-r1
dev-util/ccache:     3.1.4
dev-util/cmake:      2.8.4
sys-apps/baselayout: 1.12.14-r1
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.13::<unknown repository>, 2.65-r1
sys-devel/automake:  1.4_p6::<unknown repository>, 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1::<unknown repository>, 1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.1.2, 4.3.4, 4.4.5
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
sys-kernel/linux-headers: 2.6.36.1
virtual/os-headers:  2.6.36.1 (sys-kernel/linux-headers)
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=amdfam10 -O3 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=amdfam10 -O3 -pipe"
DISTDIR="/misc/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps y"
FEATURES="assume-digests binpkg-logs ccache distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://gentoo.virginmedia.com http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--progress --stats"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="3dnow X Xaw3d aac aalib acl acpi aim alsa amd64 apache2 audiofile avi bash-completion berkdb bluetooth bmp bonobo bzip2 caps cdb cdr cjk cli consolekit cpdflib cracklib crypt cups curl cxx dba dbm dbus dga directfb divx4linux doc dri dts dvb dvd dvdr encode esd exif expat fam fbcon fftw flac flash foomaticdb fortran ftp gd gdbm gif gnome gphoto2 gpm gstreamer gtk gtk2 gtkhtml iconv icq imagemagick imap imlib innodb ipv6 irmc jabber jadetex java jbig joystick jpeg jpeg2k kde kerberos lcms ldap libedit libwww lirc lm_sensors mad maildir mailwrapper mhash mime ming mmx mng modules mp3 mpeg msn mudflap mysql ncurses nls nptl nptlonly offensive ogg openal opengl openmp oscar pam pcre pdf pdflib perl php png postgres ppds pppd pulseaudio python qt4 quicktime readline samba sasl sdl seamonkey semantic-desktop session slp snmp spell sse sse2 ssl svg sysfs tcl tcltk tcpd tetex theora tiff tk truetype unicode usb v4l2 videos vorbis wmf xattr xine xinerama xml xml2 xorg xpm xsl xv xvid yahoo zlib" ALSA_CARDS="emu10k1 intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CAMERAS="canon" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev keyboard joystick mouse vmmouse" KERNEL="linux" LCD_DEVICES="mtxorb xosd text ncurses" LIRC_DEVICES="hauppauge" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev vesa vga intel nv nvidia mga r128 radeon vmware v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS

[Comment 1 SpanKY 2012-05-17 05:44:57 UTC]

does it still happen with newer versions ?

[Comment 2 Spooky Ghost 2012-05-17 07:55:57 UTC]

I'm on the latest stable version of nfs-utils which is still 1.2.3-r1 and it is still a problem.  I will test 1.2.5 which is the latest version in portage to see if that shows the same behaviour.

Reported by top:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10478 root      20   0 1008m 685m 1588 S    0 40.0   1:35.90 rpc.mountd

# uptime
 08:49:09 up 10 days, 20:33,  2 users,  load average: 0.05, 0.10, 0.07

# uname -r
3.2.16

# equery l net-fs/nfs-utils
 * Searching for nfs-utils in net-fs ...
[IP-] [  ] net-fs/nfs-utils-1.2.3-r1:0

# equery u nfs-utils
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for net-fs/nfs-utils-1.2.3-r1:
 U I
 + + caps     : Use Linux capabilities library to control privilege
 + + ipv6     : Adds support for IP version 6
 - - kerberos : Adds kerberos support
 + + nfsv3    : Enable support for NFSv3
 + + nfsv4    : Enable support for NFSv4
 + + tcpd     : Adds support for TCP wrappers

[Comment 3 Spooky Ghost 2012-05-24 15:49:21 UTC]

I have been testing 1.2.5 for the past few days but I see the same behaviour:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29274 root      20   0  803m 719m 2000 S  0.0 41.9   1:15.45 rpc.mountd

$ equery l net-fs/nfs-utils
 * Searching for nfs-utils in net-fs ...
[IP-] [  ] net-fs/nfs-utils-1.2.5:0

$ equery u nfs-utils
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for net-fs/nfs-utils-1.2.5:
 U I
 + + caps     : Use Linux capabilities library to control privilege
 + + ipv6     : Adds support for IP version 6
 - - kerberos : Adds kerberos support
 + + nfsidmap : Enable support for newer nfsidmap helper
 + + nfsv4    : Enable support for NFSv4
 + + nfsv41   : Enable support for NFSv4.1
 + + tcpd     : Adds support for TCP wrappers


I note that 1.2.6 has made an appearance in the portage tree so I will bump the version again to see what happens...

[Comment 4 Spooky Ghost 2012-06-20 11:34:40 UTC]

With 1.2.6 I still see the same behaviour.  After an uptime of 16 days the rpc.mountd usage is:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12188 root      20   0 1409m 569m 2216 S  0.0 33.2   2:17.56 rpc.mountd

If this is not being seen by other people I am wondering if it is being caused by my use of nss_ldap as I have also seen some problems with the memory usage of nscd.  I am going to try the latest version of nss_ldap in portage to see if there is any change in the behviour of rpc.mountd.