Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 730040 - >=dev-lang/python-3.7: Wrong install location for 64bit libraries on amd64 multilib system
Summary: >=dev-lang/python-3.7: Wrong install location for 64bit libraries on amd64 mu...
Status: RESOLVED INVALID
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-28 15:40 UTC by Thomas Sachau
Modified: 2020-07-04 12:57 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Sachau gentoo-dev 2020-06-28 15:40:18 UTC
file /usr/lib/python3.7/lib-dynload/_struct.cpython-37m-x86_64-linux-gnu.so 
/usr/lib/python3.7/lib-dynload/_struct.cpython-37m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

The same result for any other file inside that directory.

For python-3.6 i am instead having those files in /usr/lib64/python3.6/lib/dynload.

So it seems like python-3.7 and higher ignore the libdir and install 64bit files in the wrong location.

emerge -pv python:3.7:
[ebuild   R    ] dev-lang/python-3.7.7-r2:3.7/3.7m::gentoo  USE="(abiwrapper) hardened ncurses readline sqlite ssl tk xml -bluetooth -build -examples -gdbm -ipv6 -libressl -test -wininst" MULTILIB_ABI="amd64 x86" 0 KiB

emerge --info:

Portage 2.3.83-multilib (python 3.6.10-final-0, default/linux/amd64/17.1/hardened, gcc-9.2.0, glibc-2.31-r5, 4.19.124-gentoo x86_64)
=================================================================
System uname: Linux-4.19.124-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790_CPU_@_3.60GHz-with-gentoo-2.7
KiB Mem:    32856940 total,  27903940 free
KiB Swap:    1048572 total,   1048572 free
Head commit of repository gentoo: 3fb9321eeda5cdfcf6c70138699476b10a1478b8

sh bash 5.0_p17
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
app-shells/bash:          5.0_p17::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.30.3-r1::gentoo
dev-lang/python:          2.7.18::gentoo, 3.6.10-r2::gentoo, 3.7.7-r2::gentoo, 3.8.3::gentoo
dev-util/cmake:           3.17.3::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.20::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.13.4-r2::gentoo, 1.15.1-r2::gentoo, 1.16.2::gentoo
sys-devel/binutils:       2.32-r1::gentoo, 2.33.1-r1::gentoo, 2.34-r1::gentoo
sys-devel/gcc:            8.2.0-r6::gentoo, 9.2.0-r4::gentoo, 10.1.0-r1::gentoo
sys-devel/gcc-config:     2.3::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.7::gentoo (virtual/os-headers)
sys-libs/glibc:           2.31-r5::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: git
    sync-uri: git+ssh://git@git.gentoo.org/repo/gentoo.git
    priority: -1000
    eclass-overrides: gentoo local

eclipse
    location: /var/lib/layman/eclipse
    masters: gentoo
    priority: 50
    eclass-overrides: gentoo local

enlightenment
    location: /var/lib/layman/enlightenment
    masters: gentoo
    priority: 50
    eclass-overrides: gentoo local

java
    location: /var/lib/layman/java
    masters: gentoo
    priority: 50
    eclass-overrides: gentoo local

multilib-portage
    location: /var/lib/layman/multilib-portage
    masters: gentoo
    priority: 50
    eclass-overrides: gentoo local

tastytea
    location: /var/lib/layman/tastytea
    masters: gentoo
    priority: 50
    eclass-overrides: gentoo local

local
    location: /usr/local/portage
    masters: gentoo
    priority: 100
    eclass-overrides: gentoo local

Installed sets: @fonts
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs collision-protect distlocks ebuild-locks fixlafiles force-multilib ipc-sandbox merge-sync metadata-transfer network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms sign splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp.snt.utwente.nl/pub/os/linux/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,--as-needed -Wl,--hash-style=gnu"
LINGUAS="en de"
MAKEOPTS="-j6 --load-average=8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow X alsa amd64 crypt cups custom-cflags custom-cxxflags custom-optimization gpm hardened java5 java6 libtirpc mmx ncurses nls nptl nsplugin ogg openmp opus pam pie readline scanner seccomp split-usr sse sse2 ssl ssp system-sqlite unicode v4l vorbis vulkan xattr xinerama xtpax zlib" ADA_TARGET="gnat_2018" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en-GB de" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" MULTILIB_ABI="amd64 x86" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7 python3_6" RUBY_TARGETS="ruby26" SANE_BACKENDS="hp hp3500 hp3900 hp4200 hp5400 hp5590 hpljm1005 hpsj5s hs2p" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-06-28 15:45:33 UTC
The location is not wrong.  It's permitted to install host-native executables in subdirs of /usr/lib.  This is what upstream is doing, this is what other upstreams roughly expect and no, we're not going to continue carrying a patchset just because in 2000s someone suddenly started moving everything to lib64 because they could.
Comment 2 Mike Gilbert gentoo-dev 2020-06-28 16:27:01 UTC
To expand on mgorny's response:

This was an intentional decision made by the python team primarily to reduce the amount of distro-specific patching we need to do.

It may also be helpful when cross-compiling modules if the python libdir is consistent across all archs.
Comment 3 Thomas Sachau gentoo-dev 2020-07-04 08:05:08 UTC
So lets first start with the official docs:

https://wiki.gentoo.org/wiki/Project:AMD64/Multilib_layout says:

Any ebuilds that attempt to install native libraries to /usr/lib need to be fixed to use get_libdir function (provided by EAPI 6 or multilib.eclass in earlier EAPIs) to obtain the correct library directory name.

And those files inside /usr/lib/python3.7/lib-dynload are obviously native libraries.

So per the official docs, those should instead go to /usr/lib64/python3.7/lib-dynload.

The answer from mgorny is about "host-native executables", which is not the same as native libraries. executables should usually go to /usr/bin. Not sure, why you would want to install them into /usr/lib. Installing them into a fixed none abi-specific location will prevent people from cross-compiling/creating none host-native executables as the install location is fixed in the ebuild. So this would either overwrite the host-native executable or getting overwritten by the host-native executable.

For cross-compiling: Not sure, where you got the info, that the current layout helps cross-compiling as it is the exact opposite. If abi-specific files are inside a fixed none abi-specific directory (like the fixed /usr/lib/python3.7 in this case), you cannot install abi-specific files for all ABIs (like for example 32bit libs and 64bit libs on amd64 multilib). But without 32bit libs avaidable, you cannot cross-compile and link anything that wants to use those 32bit libs.
Comment 4 Mike Gilbert gentoo-dev 2020-07-04 12:54:18 UTC
(In reply to Thomas Sachau from comment #3)
> So lets first start with the official docs:
> 
> https://wiki.gentoo.org/wiki/Project:AMD64/Multilib_layout says:
> 
> Any ebuilds that attempt to install native libraries to /usr/lib need to be
> fixed to use get_libdir function (provided by EAPI 6 or multilib.eclass in
> earlier EAPIs) to obtain the correct library directory name.
> 
> And those files inside /usr/lib/python3.7/lib-dynload are obviously native
> libraries.
> 
> So per the official docs, those should instead go to
> /usr/lib64/python3.7/lib-dynload.

There is no mention of subdirectories of /usr/lib in the documentation you quoted.

> For cross-compiling: Not sure, where you got the info, that the current
> layout helps cross-compiling as it is the exact opposite. If abi-specific
> files are inside a fixed none abi-specific directory (like the fixed
> /usr/lib/python3.7 in this case), you cannot install abi-specific files for
> all ABIs (like for example 32bit libs and 64bit libs on amd64 multilib). But
> without 32bit libs avaidable, you cannot cross-compile and link anything
> that wants to use those 32bit libs.

I mean cross-compiling where ROOT != /, not multilib for the same with the same ROOT.

When cross-compiling, you generally have to invoke CBUILD python to install modules for CHOST, and it is helpful if both pythons install things into the same site-packages directory. Collisions are avoided by use of the ROOT variable.

Regarding multilib:

1. We don't support multilib Python in Gentoo, and have no plans to do so.

2. In modern versions of python, extension modules have the ABI encoded into the filename (cpython-37m-x86_64-linux-gnu), which ensures that installing multiple ABIs in /usr/lib/python will not cause collisions.
Comment 5 Mike Gilbert gentoo-dev 2020-07-04 12:57:33 UTC
Here is the current policy on installing things in /usr/lib vs /usr/lib64.

https://projects.gentoo.org/qa/policy-guide/filesystem.html#pg0203

"Libraries installed as a part of larger software package can be installed along with it into a subdirectory of lib."