Bug 448858 – sys-libs/glibc-2.17 - python2.7: relocation error: /lib/libresolv.so.2: symbol __sendmmsg, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Toralf Förster 2012-12-27 15:38:54 UTC]

Created attachment 333478 [details]
sys-libs:glibc-2.17:20121227-142120.log.gz

At an unstable Gentoo Linux (chrooted system) I got this :

# emerge -au glibc

These are the packages that would be merged, in reverse order:

Calculating dependencies  . ... done!
[ebuild     U *] sys-libs/glibc-2.17 [2.16.0]

Would you like to merge these packages? [Yes/No] 
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) sys-libs/glibc-2.17
>>> Installing (1 of 1) sys-libs/glibc-2.17
>>> Recording sys-libs/glibc in "world" favorites file...
>>> Jobs: 0 of 1 complete                           Load avg: 7.4, 7.7, 17.8/usr/bin/python2.7: relocation error: /lib/libresolv.so.2: symbol __sendmmsg, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference              



n22 ~ # emerge --info glibc                                                                           
Portage 2.1.11.38 (default/linux/x86/10.0, gcc-4.7.2, glibc-2.17, 3.7.1 i686)                         
=================================================================                                     
                        System Settings                                                               
=================================================================                                     
System uname: Linux-3.7.1-i686-Intel-R-_Core-TM-_i5-2540M_CPU_@_2.60GHz-with-gentoo-2.2               
Timestamp of tree: Thu, 27 Dec 2012 13:15:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p39-r1
dev-lang/python:          2.7.3-r3, 3.2.3-r2
dev-util/cmake:           2.8.10.2
dev-util/pkgconfig:       0.27.1
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6
sys-devel/autoconf:       2.69
sys-devel/automake:       1.12.6
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.7.2
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo toralf
ACCEPT_KEYWORDS="x86 ~x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--autounmask=n --keep-going=y --nospinner --tree --quiet-build --deep"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-logs compress-build-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict test test-fail-continue unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acl apache2 berkdb bzip2 cli cracklib crypt cups cxx dri fam fastbuild gdbm gmp gpm iconv ipv6 logrotate mmx modules mudflap mysql mysqli ncurses nls nptl openmp pam pcre pppd readline session sse sse2 sse3 ssl ssse3 tcpd threads unicode userlocales webmail x86 xml zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="access actions alias auth_basic auth_digest authn_anon authn_core authn_dbd authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi compat dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

sys-libs/glibc-2.17 was built with the following:
USE="test -debug -gd (-hardened) (-multilib) -profile (-selinux) -systemtap -vanilla"
CFLAGS="-march=native -pipe -g -ggdb -O2 -fno-strict-aliasing"
CXXFLAGS="-march=native -pipe -g -ggdb -O2 -fno-strict-aliasing"

[Comment 1 SpanKY 2012-12-27 21:41:38 UTC]

what do you see if you run the commands as shown below ?

$ file -L /lib/libc.so.6
/lib/libc.so.6: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped

$ readelf -sW /lib/libc.so.6 | grep sendmmsg
  1158: 000f58e0   168 FUNC    GLOBAL DEFAULT   11 __sendmmsg@@GLIBC_PRIVATE
  1187: 000f58e0   168 FUNC    WEAK   DEFAULT   11 sendmmsg@@GLIBC_2.14

$ readelf -sW /lib/libresolv.so.2 | grep sendmmsg
    45: 00000000     0 FUNC    GLOBAL DEFAULT  UND __sendmmsg@GLIBC_PRIVATE (9)

$ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.7.2.
Compiled on a Linux 3.7.0 system on 2012-12-25.
Available extensions:
        C stubs add-on version 2.1.2
        crypt add-on version 2.1 by Michael Glad and others
        Gentoo patchset 1
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

[Comment 2 SpanKY 2012-12-27 21:48:02 UTC]

hmm, actually that was with a 32bit multilib.  i just built 32bit native and see slightly different output:

$ readelf -sW /lib/libc.so.6 | grep sendmmsg
  1181: 000eb4a0   192 FUNC    GLOBAL DEFAULT   11 sendmmsg@@GLIBC_2.14

$ readelf -sW /lib/libresolv.so.2 | grep sendmmsg
    38: 00000000     0 FUNC    GLOBAL DEFAULT  UND sendmmsg@GLIBC_2.14 (13)

it's a little disconcerting that the native 32bit is diff from the multilib 32bit

[Comment 3 Toralf Förster 2012-12-27 21:59:09 UTC]

Here're my outputs :

tfoerste@n22 ~/devel/linux $ sudo ~/workspace/bin/chr_uml.sh -r /home/tfoerste/virtual/uml/n22unst4 
n22 ~ # file -L /lib/libc.so.6
/lib/libc.so.6: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped                                                          
n22 ~ #                                                                                               
n22 ~ # readelf -sW /lib/libc.so.6 | grep sendmmsg                                                    
  1158: 000f8410   192 FUNC    GLOBAL DEFAULT   12 __sendmmsg@@GLIBC_PRIVATE
  1187: 000f8410   192 FUNC    WEAK   DEFAULT   12 sendmmsg@@GLIBC_2.14
n22 ~ #                                                                                               
n22 ~ # readelf -sW /lib/libresolv.so.2 | grep sendmmsg                                               
    45: 00000000     0 FUNC    GLOBAL DEFAULT  UND __sendmmsg@GLIBC_PRIVATE (9)
n22 ~ #                                                                                               
n22 ~ # /lib/libc.so.6                                                                                
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.7.2.
Compiled on a Linux 3.7.0 system on 2012-12-27.
Available extensions:
        C stubs add-on version 2.1.2
        crypt add-on version 2.1 by Michael Glad and others
        Gentoo patchset 1
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

[Comment 4 SpanKY 2012-12-27 22:04:54 UTC]

(In reply to comment #3)

looks to me like the library is correct.  maybe python had libc.so loaded in memory, updated glibc, then invoked some code where it needed to load libresolv.so dynamically ?

does python still fail ?  i.e. can you do `emerge pax-utils portage-utils` (or some other simple packages) and have it work ?

[Comment 5 Toralf Förster 2012-12-27 22:13:12 UTC]

(In reply to comment #4)
> (In reply to comment #3)
> 
> looks to me like the library is correct.  maybe python had libc.so loaded in
> memory, updated glibc, then invoked some code where it needed to load
> libresolv.so dynamically ?
yes - I think too

> does python still fail ?  i.e. can you do `emerge pax-utils portage-utils`
> (or some other simple packages) and have it work ?

no - works fine so far.

[Comment 6 SpanKY 2012-12-28 16:50:37 UTC]

looks like bad interaction with portage then when upgrading glibc

i'd hate to think we'd have to mark the system C library as a reload check point like we do with portage itself, but maybe we don't have a choice :(

[Comment 7 Toralf Förster 2012-12-28 17:03:25 UTC]

FWIW the package itself was installed fine, but a lock file wasn't removed from the /var/tmp/portage directory IIRC

[Comment 8 Zac Medico 2012-12-28 22:10:30 UTC]

(In reply to comment #4)
> looks to me like the library is correct.  maybe python had libc.so loaded in
> memory, updated glibc, then invoked some code where it needed to load
> libresolv.so dynamically ?

It could be the code from bug #439584, which uses ctypes to load libc.so into memory, and caches it:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=978f3c6d514f0fcf9329d536cc43cf1230e23112
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=10b6d0129d062f4d5d8a7611023c3f8cc43f1eab

[Comment 9 Zac Medico 2012-12-28 22:32:11 UTC]

Here's a patch to disable caching of libraries:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9e37cca4f54260bd8c45a3041fcee00938c71649

[Comment 10 SpanKY 2012-12-29 04:41:44 UTC]

(In reply to comment #9)

how about keeping the handle cached, but check things like its stat date ?  look at mtime/ctime/size/inode ?

[Comment 11 Zac Medico 2012-12-29 04:46:55 UTC]

(In reply to comment #10)
> (In reply to comment #9)
> 
> how about keeping the handle cached, but check things like its stat date ? 
> look at mtime/ctime/size/inode ?

I thought about that, but I didn't see a convenient way to get the exact path of the library file, since ctypes.util.find_library('c') only returns 'libc.so.6'.

Maybe it's not worth caching the handle anyway.

[Comment 12 Kacper Kowalik (Xarthisius) (RETIRED) 2012-12-29 10:25:39 UTC]

(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > 
> > how about keeping the handle cached, but check things like its stat date ? 
> > look at mtime/ctime/size/inode ?
> 
> I thought about that, but I didn't see a convenient way to get the exact
> path of the library file, since ctypes.util.find_library('c') only returns
> 'libc.so.6'.

How about parsing output of '/sbin/ldconfig -p'? That's one of the mechanisms utilized by find_library too.

> Maybe it's not worth caching the handle anyway.

[Comment 13 Zac Medico 2012-12-29 21:23:17 UTC]

(In reply to comment #12)
> How about parsing output of '/sbin/ldconfig -p'? That's one of the
> mechanisms utilized by find_library too.

I'd prefer to let python's ctypes api handle the parsing, if possible. Anyway, the c library seems to load fairly quickly.

[Comment 14 Arfrever Frehtes Taifersar Arahesis 2012-12-30 04:33:17 UTC]

_ctypes internally uses C dlopen(), which has its own cache of library handles. Deletion of ctypes.CDLL objects does not trigger call to C dlclose(). Direct call to _ctypes.dlclose() would be needed. (See also http://bugs.python.org/issue14597.)

# python
Python 3.3.0+ (3.3:44a4f9289faa+, Dec 30 2012, 03:50:43) 
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes, _ctypes, os
>>> os.system("gcc -xc - -fPIC -shared -o /usr/lib64/libtest.so <<< 'void func1(){}'")
0
>>> l = ctypes.cdll.LoadLibrary("libtest.so")
>>> l
<CDLL 'libtest.so', handle 1b4f8f0 at 7f9f8e9b7250>
>>> l._handle
28637424
>>> l.func1
<_FuncPtr object at 0x7f9f8ea5b6d0>
>>> del l
>>> _ctypes.dlclose(28637424)
>>> _ctypes.dlclose(28637424)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: ion: shared object not open
>>> l = ctypes.cdll.LoadLibrary("libtest.so")
>>> l
<CDLL 'libtest.so', handle 1b4f8f0 at 7f9f8ea26b50>
>>> l._handle
28637424
>>> l.func1
<_FuncPtr object at 0x7f9f8ea5b7a0>
>>> os.system("gcc -xc - -fPIC -shared -o /usr/lib64/libtest.so <<< 'void func2(){}'")
0
>>> l = ctypes.cdll.LoadLibrary("libtest.so")
>>> l
<CDLL 'libtest.so', handle 1b4f8f0 at 7f9f8ea26c50>
>>> l._handle
28637424
>>> l.func1
<_FuncPtr object at 0x7f9f8ea5b870>
>>> l.func2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/ctypes/__init__.py", line 366, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib64/python3.3/ctypes/__init__.py", line 371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib64/libtest.so: undefined symbol: func2
>>> _ctypes.dlclose(28637424)
>>> _ctypes.dlclose(28637424)
>>> _ctypes.dlclose(28637424)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError
>>> l = ctypes.cdll.LoadLibrary("libtest.so")
>>> l
<CDLL 'libtest.so', handle 1b50350 at 7f9f8ea26d10>
>>> l._handle
28640080
>>> l.func1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/ctypes/__init__.py", line 366, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib64/python3.3/ctypes/__init__.py", line 371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib64/libtest.so: undefined symbol: func1
>>> l.func2
<_FuncPtr object at 0x7f9f8ea5b940>

[Comment 15 Zac Medico 2012-12-30 07:52:30 UTC]

(In reply to comment #14)
> _ctypes internally uses C dlopen(), which has its own cache of library
> handles. Deletion of ctypes.CDLL objects does not trigger call to C
> dlclose(). Direct call to _ctypes.dlclose() would be needed. (See also
> http://bugs.python.org/issue14597.)

We could just ignore this for now, since when emerge updates packages, the only  library that's loaded is always in a subprocess which forked from MergeProcess. That subprocess exits soon after loading the library, and the dlopen cache will disappear with it.

[Comment 16 Zac Medico 2012-12-30 08:05:55 UTC]

Created attachment 333736 [details, diff]
for demonstration only, add proxy to call dlclose from __del__

This patch demonstrates that we could use a proxy to call dlclose automatically.

Here's a demonstration showing that if we don't hold a reference to the library, it will be deleted when _get_syncfs() returns. Even after dlclose is called by __del__, it seems that it's still possible to successfully call the syncfs _FuncPtr object (though it's probably not very safe practice). In practice, we wouldn't want to call dlclose until after we've called the syncfs.

Python 2.7.3 (default, May  5 2012, 23:40:16) 
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from portage.dbapi.vartree import _get_syncfs
>>> syncfs = _get_syncfs()
_LibraryHandleProxy __del__ 4152121656
>>> syncfs
<_FuncPtr object at 0xa01b1cc>
>>> f = open('/tmp/foo', 'wb')
>>> syncfs(f.fileno())
0

[Comment 17 Zac Medico 2012-12-30 09:37:57 UTC]

For additional safety, I've added an additional fork for every time that we load a library with ctypes:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=7ebb2f54877edb28621c33e380f8777b1b1dc201

[Comment 18 SpanKY 2012-12-30 17:50:32 UTC]

(In reply to comment #17)

can't we do something more pythonic ?  like attempt an operation and if it fails, catch the exception and retry ?  this error case happens very infrequently.

[Comment 19 Zac Medico 2012-12-30 22:09:37 UTC]

(In reply to comment #18)

Experience seems to indicate that ctypes is prone to putting the process in an irrecoverable state (leading to a segfault, relocation error, or who knows what). For things that are fragile like this, I typically isolate them in a subprocess in order to protect the main process from corruption. For some kinds corruption, there's no other good way to recover that I'm aware of.

[Comment 20 Zac Medico 2013-01-10 15:37:32 UTC]

This is fixed in 2.1.11.39 and 2.2.0_alpha150.

[Comment 21 SpanKY 2013-04-02 01:09:27 UTC]

*** Bug 464104 has been marked as a duplicate of this bug. ***

[Comment 22 Martin von Gagern 2013-04-03 05:30:43 UTC]

(In reply to comment #20)
> This is fixed in 2.1.11.39 and 2.2.0_alpha150.

Note that bug #464104, which was marked a duplicate of this one here, was reported against portage 2.2.0_alpha169. So it seems the fix either didn't work or there was a regression reintroducing the issue.

[Comment 23 Zac Medico 2013-04-03 05:50:42 UTC]

(In reply to comment #22)
> (In reply to comment #20)
> > This is fixed in 2.1.11.39 and 2.2.0_alpha150.
> 
> Note that bug #464104, which was marked a duplicate of this one here, was
> reported against portage 2.2.0_alpha169. So it seems the fix either didn't
> work or there was a regression reintroducing the issue.

It seems like a different code path is triggering bug 464104, so I've re-opened it for investigation.