Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 692128 - dev-lang/python-3.6.? USE=ncurses with sys-libs/ncurses[tinfo] - .../work/Python-3.6.5/python: symbol 'LINES': can't resolve symbol
Summary: dev-lang/python-3.6.? USE=ncurses with sys-libs/ncurses[tinfo] - .../work/Pyt...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Python Gentoo Team
URL: https://bugs.python.org/issue38318
Whiteboard:
Keywords:
Depends on:
Blocks: tinfo
  Show dependency tree
 
Reported: 2019-08-14 10:49 UTC by David Flogeras
Modified: 2019-11-20 16:32 UTC (History)
8 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
proof-of-concept patch (python-3.6.5-uclibc-tinfo-link.patch,626 bytes, patch)
2019-08-14 10:50 UTC, David Flogeras
Details | Diff
failed build log as requested (build.log.xz,33.46 KB, application/x-xz)
2019-08-16 20:08 UTC, David Flogeras
Details
build.log from armv7a catalyst stage3 build (file_692128.txt,118.01 KB, text/plain)
2019-09-09 20:31 UTC, Michael 'veremitz' Everitt
Details
PoC patch python2.7 (file_692128.txt,606 bytes, text/plain)
2019-10-01 23:12 UTC, Michael 'veremitz' Everitt
Details
PoC patch python3.6 (file_692128.txt,548 bytes, text/plain)
2019-10-01 23:29 UTC, Michael 'veremitz' Everitt
Details
Python_2.7 patch take 2 (file_692128.txt,1.03 KB, patch)
2019-10-02 23:39 UTC, Michael 'veremitz' Everitt
Details | Diff
Python_3.6 patch take 2 (file_692128.txt,1.24 KB, patch)
2019-10-02 23:42 UTC, Michael 'veremitz' Everitt
Details | Diff
Failed build log with latest patch on python:3.6 (build.log,686.37 KB, text/plain)
2019-10-03 11:40 UTC, David Flogeras
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Flogeras 2019-08-14 10:49:49 UTC
This seems to be because python is solely linking to ncurses[w].  I believe it needs also to link to tinfo.

See also:
https://bugs.gentoo.org/690222

I'm not sure what the proper fix is because I don't fully understand python's build system, but I'll attach a proof-of-concept patch that at least got my system to compile python successfully.


Reproducible: Always
Comment 1 David Flogeras 2019-08-14 10:50:18 UTC
Created attachment 586782 [details, diff]
proof-of-concept patch
Comment 2 Jeroen Roovers gentoo-dev 2019-08-16 10:25:39 UTC
Please attach the entire build log to this bug report.
Please post your `emerge --info` output in a comment.
Comment 3 Jeroen Roovers gentoo-dev 2019-08-16 10:28:47 UTC
Comment on attachment 586782 [details, diff]
proof-of-concept patch

setup.py should check whether terminal info specific functions are provided by libncurses or by libtinfo, instead of hard-coding it as in your patch.
Comment 4 David Flogeras 2019-08-16 19:00:27 UTC
I apologize, I completely failed to mention this is related to uclibc-ng, and blueness is in on it.

Here's emerge --info, build log to follow once the system can produce it

emerge --info
Portage 2.3.69 (python 3.6.5-final-0, default/linux/arm/17.0/uclibc/armv6j, gcc-8.3.0, uclibc-ng-1.0.31, 4.19.66-gentoo armv6l)
=================================================================
System uname: Linux-4.19.66-gentoo-armv6l-Intel-R-_Core-TM-_i7-4770_CPU_@_3.40GHz-with-gentoo-2.6
KiB Mem:    16336000 total,   6210652 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Thu, 15 Aug 2019 10:15:01 +0000
Head commit of repository gentoo: 9a1a265385cfcf00088a15ba81d54a791a504ce5
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.28.2-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.14.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.41.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.31.1-r6::gentoo, 2.32-r1::gentoo
sys-devel/gcc:            8.3.0-r1::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.19::gentoo (virtual/os-headers)
sys-libs/uclibc-ng:       1.0.31::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: no
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="arm"
ACCEPT_LICENSE="@FREE"
CBUILD="armv6j-unknown-linux-uclibceabihf"
CFLAGS="-march=armv6j -mfpu=vfp -mfloat-abi=hard -pipe -O2"
CHOST="armv6j-unknown-linux-uclibceabihf"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=armv6j -mfpu=vfp -mfloat-abi=hard -pipe -O2"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe -march=armv6j -mfpu=vfp -mfloat-abi=hard"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news nodoc noinfo noman parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv6j -mfpu=vfp -mfloat-abi=hard"
GENTOO_MIRRORS="http://mirror.its.dal.ca/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en"
MAKEOPTS="-j8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl arm bzip2 cli crypt cxx dri gdbm iconv ncurses nptl openmp pcre readline seccomp split-usr ssl tcpd uclibc unicode xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="uclibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" L10N="en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="exynos fbdev omap dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 5 David Flogeras 2019-08-16 20:08:43 UTC
Created attachment 587070 [details]
failed build log as requested
Comment 6 Michael 'veremitz' Everitt 2019-09-09 00:55:12 UTC
Also affecting me on python 2.7.10 ...

build.log to follow.
Comment 7 Michael 'veremitz' Everitt 2019-09-09 20:31:20 UTC
Created attachment 589594 [details]
build.log from armv7a catalyst stage3 build
Comment 8 Mike Gilbert gentoo-dev 2019-09-10 02:27:50 UTC
Comment on attachment 587070 [details]
failed build log as requested

Please do not compress build logs under 1 MB. It just makes it harder to view them in a web browser.
Comment 9 Mike Gilbert gentoo-dev 2019-09-10 02:30:44 UTC
I am unable to produce either failure on amd64/glibc.
Comment 10 Michael 'veremitz' Everitt 2019-09-10 05:28:40 UTC
(In reply to Mike Gilbert from comment #9)
> I am unable to produce either failure on amd64/glibc.

(In reply to David Flogeras from comment #4)
> I apologize, I completely failed to mention this is related to uclibc-ng,
> and blueness is in on it.
> 
> sys-libs/uclibc-ng:       1.0.31::gentoo

This is a peculiarity for uclibc, it doesn't even manifest on musl.

Further investigation tends to suggest it's missing -ltinfow in addition to -lncursesw in LDFLAGS possibly? There is a patch for pkg-config but it would appear that it's still not using that for ncurses link detection somewhere/somehow.
Comment 11 Michael 'veremitz' Everitt 2019-09-10 05:36:05 UTC
(In reply to Michael 'veremitz' Everitt from comment #10)
> This is a peculiarity for uclibc, it doesn't even manifest on musl.
> 
> Further investigation tends to suggest it's missing -ltinfow in addition to
> -lncursesw in LDFLAGS possibly? There is a patch for pkg-config but it would
> appear that it's still not using that for ncurses link detection
> somewhere/somehow.

ie. eg. http://dpaste.com/3SP0BDV#line-941 (also in build log, paste will expire)
Comment 12 Mike Gilbert gentoo-dev 2019-09-10 18:20:53 UTC
(In reply to Michael 'veremitz' Everitt from comment #11)

It does seem like -ltinfo or -ltinfow is missing, but I'm baffled as to why it doesn't show up on glibc.
Comment 13 Martin Väth 2019-09-29 06:17:50 UTC
Here, the problem shows up on glibc (amd64 and x86) with all 3.6 and 3.7 python versions up to (including) 3.6.9 and 3.7.4-r1
Comment 14 Martin Väth 2019-09-29 07:01:26 UTC
Note: The patch from comment #1 fixes the problem.

IMHO, this patch should not be modified to dynamically detect the library:
This would break if the user compiles ncurses with a toggled tinfo.
So IMHO, what is needed is instead a new USE-flag tinfo with a proper USE-depend and a conditional application of the patch.
Comment 15 Michael 'veremitz' Everitt 2019-09-29 20:07:43 UTC
(In reply to Martin Väth from comment #14)
> Note: The patch from comment #1 fixes the problem.
> 
> IMHO, this patch should not be modified to dynamically detect the library:
> This would break if the user compiles ncurses with a toggled tinfo.
> So IMHO, what is needed is instead a new USE-flag tinfo with a proper
> USE-depend and a conditional application of the patch.

Should be feasible with a conditional based on eg. has_version ncurses[tinfo] perhaps .. ?
Comment 16 Martin Väth 2019-09-29 20:16:08 UTC
(In reply to Michael 'veremitz' Everitt from comment #15)
> has_version ncurses[tinfo]

No, since this is practically the same as compile-time testing, only.

For instance, if a user adds
sys-libs/ncurses tinfo
to package.use and then runs emerge -NDu @world, then only ncurses will be recompiled, but portage will not recompile python and will not even warn that python will be broken afterwards.
Comment 17 Michael 'veremitz' Everitt 2019-09-29 20:18:17 UTC
(In reply to Martin Väth from comment #16)
> (In reply to Michael 'veremitz' Everitt from comment #15)
> > has_version ncurses[tinfo]
> 
> No, since this is practically the same as compile-time testing, only.
> 
> For instance, if a user adds
> sys-libs/ncurses tinfo
> to package.use and then runs emerge -NDu @world, then only ncurses will be
> recompiled, but portage will not recompile python and will not even warn
> that python will be broken afterwards.

Mmm, fair point .. but there should be some link between the ncurses Python USE flag then .. 

Outta my dept here now .. but I see your point! :)
Comment 18 Mike Gilbert gentoo-dev 2019-09-29 21:29:26 UTC
We need someone to submit a patch uptream on this. Happy to backport from there.
Comment 19 Michael 'veremitz' Everitt 2019-09-30 01:54:05 UTC
(In reply to Mike Gilbert from comment #18)
> We need someone to submit a patch uptream on this. Happy to backport from
> there.

I will take this up then.

also of interest to some, konsolebox's comment at https://bugs.gentoo.org/457530#c20 for 'matching' USE flag/rebuild scenarios.
Comment 20 Michael 'veremitz' Everitt 2019-09-30 02:40:37 UTC
Filed https://bugs.python.org/issue38318 .
Comment 21 Michael 'veremitz' Everitt 2019-10-01 23:12:19 UTC
Created attachment 591532 [details]
PoC patch python2.7

This is a slightly less-crude compile-time patch which works for me for Python2.7. Will adjust shortly for 3.x
Comment 22 Michael 'veremitz' Everitt 2019-10-01 23:29:25 UTC
Created attachment 591538 [details]
PoC patch python3.6

Similarly crude compile-time patch for python3.6.

I'm deliberately ignoring the separate issues of recompiling python when [tinfo] is toggled, and any issue that may be arising from linker configuration options that may be enabled/disabled for uclibc-ng on my local x86_64/ARM systems at this point.

These patches /shouldn't/ affect these scenarios, nor glibc, as far as I can tell however, although will await review.
Comment 23 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-10-02 02:42:47 UTC
(In reply to Michael 'veremitz' Everitt from comment #21)
> Created attachment 591532 [details]
> PoC patch python2.7
> 
> This is a slightly less-crude compile-time patch which works for me for
> Python2.7. Will adjust shortly for 3.x

That doesn't account for tinfow.
Comment 24 Michael 'veremitz' Everitt 2019-10-02 09:06:56 UTC
(In reply to Michał Górny from comment #23)
> (In reply to Michael 'veremitz' Everitt from comment #21)
> > Created attachment 591532 [details]
> > PoC patch python2.7
> > 
> > This is a slightly less-crude compile-time patch which works for me for
> > Python2.7. Will adjust shortly for 3.x
> 
> That doesn't account for tinfow.

That's a fair assertion.. I will further re-work the logic a bit, and check that works too.
Comment 25 Michael 'veremitz' Everitt 2019-10-02 23:39:26 UTC
Created attachment 591632 [details, diff]
Python_2.7 patch take 2

Builds correctly on x86_64 with uclibc-ng, and on ARMv6zk (also with uclibc-ng.
Comment 26 Michael 'veremitz' Everitt 2019-10-02 23:42:13 UTC
Created attachment 591634 [details, diff]
Python_3.6 patch take 2

Corresponding patch for v3.6
Comment 27 David Flogeras 2019-10-03 11:38:27 UTC
Testing on armv6j-unknown-linux-uclibceabihf (rpi).

python:2.7 compiled with above patch, and import curses works as expected.

Trying with python:3.6, and I got a failure I can't understand.  Attaching build.log, emerge --info as above (same system).
Comment 28 David Flogeras 2019-10-03 11:40:03 UTC
Created attachment 591682 [details]
Failed build log with latest patch on python:3.6
Comment 29 Michael 'veremitz' Everitt 2019-10-03 12:44:09 UTC
(In reply to David Flogeras from comment #27)
> Testing on armv6j-unknown-linux-uclibceabihf (rpi).
> 
> python:2.7 compiled with above patch, and import curses works as expected.
> 
> Trying with python:3.6, and I got a failure I can't understand.  Attaching
> build.log, emerge --info as above (same system).

It doesn't /look/ like a build system failure .. but I'll compare my logs later and see if I spot anything.
If you repeat the build does it halt at the same point?
Comment 30 David Flogeras 2019-10-03 14:31:25 UTC
Hmm, nm.  Emerging again and it finished, probably unrelated to this.

In any case, python3 with above patch works for me, including importing curses module.  Thanks for the effort!
Comment 31 Michael 'veremitz' Everitt 2019-10-03 20:45:18 UTC
(In reply to David Flogeras from comment #30)
> Hmm, nm.  Emerging again and it finished, probably unrelated to this.
> 
> In any case, python3 with above patch works for me, including importing
> curses module.  Thanks for the effort!

Super, thanks for the verification! :] enjoy!
Comment 32 Steffen Christgau 2019-11-06 09:29:52 UTC
I can confirm that the two patches work on a i586-gentoo-linux-uclibc system with the default/linux/x86/17.0/uclibc/hardened profile
Comment 33 Michael 'veremitz' Everitt 2019-11-09 23:05:19 UTC
Patches from #c25 and #c26 apply and build correctly with python-2.7.16 and python-3.6.9 respectively, as recently stabilised, FYI.