Summary: | dev-python/cryptography-40.0.1: rust extension broken with LTO | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Paolo Pedroni <paolo.pedroni> |
Component: | Current packages | Assignee: | Python Gentoo Team <python> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | audvare, jstein, matoro_gentoo, mgorny |
Priority: | Normal | Keywords: | TESTFAILURE |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: | https://github.com/pyca/cryptography/issues/9023 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 618550 | ||
Attachments: | cryptography-40.0.1:20230406-115120.log.gz |
Description
Paolo Pedroni
2023-04-06 11:57:12 UTC
*** Bug 903915 has been marked as a duplicate of this bug. *** readelf -a /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1-python3_10/install/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so | grep PyInit Could it be due to this: /usr/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'cryptography.hazmat.bindings._rust' as data is deprecated, please list it in `packages`. !! ############################ # Package would be ignored # ############################ Python recognizes 'cryptography.hazmat.bindings._rust' as an importable package, but it is not listed in the `packages` configuration of setuptools. 'cryptography.hazmat.bindings._rust' has been automatically added to the distribution only because it may contain data files, but this behavior is likely to change in future versions of setuptools (and therefore is considered deprecated). Please make sure that 'cryptography.hazmat.bindings._rust' is included as a package by using the `packages` configuration field or the proper discovery methods (for example by using `find_namespace_packages(...)`/`find_namespace:` instead of `find_packages(...)`/`find:`). You can read more about "package discovery" and "data files" on setuptools documentation page. !! check.warn(importable) /usr/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'cryptography.hazmat.bindings._rust.openssl' as data is deprecated, please list it in `packages`. !! ############################ # Package would be ignored # ############################ Python recognizes 'cryptography.hazmat.bindings._rust.openssl' as an importable package, but it is not listed in the `packages` configuration of setuptools. 'cryptography.hazmat.bindings._rust.openssl' has been automatically added to the distribution only because it may contain data files, but this behavior is likely to change in future versions of setuptools (and therefore is considered deprecated). Please make sure that 'cryptography.hazmat.bindings._rust.openssl' is included as a package by using the `packages` configuration field or the proper discovery methods (for example by using `find_namespace_packages(...)`/`find_namespace:` instead of `find_packages(...)`/`find:`). You can read more about "package discovery" and "data files" on setuptools documentation page. !! check.warn(importable) Now I have plenty of python modules failing with /usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: PyInit__openssl You didn't answer my question. (In reply to Michał Górny from comment #4) > You didn't answer my question. Which question? Oh, I see. Sorry, I didn't undestand it was a question. Here it is: # readelf -a /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1-python3_10/install/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so | grep PyInit 0000001d3780 003e00000006 R_X86_64_GLOB_DAT 0000000000000000 PyInit__openssl + 0 62: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND PyInit__openssl 168: 00000000000b43a0 526 FUNC GLOBAL DEFAULT 15 PyInit__rust 5461: 00000000000b43a0 526 FUNC GLOBAL DEFAULT 15 PyInit__rust 5519: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND PyInit__openssl Now this is weird. I only have: 732: 0000000000098ae0 526 FUNC GLOBAL DEFAULT 11 PyInit__rust Feel like reporting it upstream? I'm afraid it's beyond my skills to figure out what Rust is doing here. (In reply to Michał Górny from comment #7) > Feel like reporting it upstream? I'm afraid it's beyond my skills to figure > out what Rust is doing here. If it's beyond your skill, it's certainly beyond mine as well, but I'll try... (In reply to Michał Górny from comment #7) > Now this is weird. I only have: > > 732: 0000000000098ae0 526 FUNC GLOBAL DEFAULT 11 PyInit__rust > > Feel like reporting it upstream? I'm afraid it's beyond my skills to figure > out what Rust is doing here. Which version of openssl do you have? I'm trying openssl-3 ATM. Could it be the culprit? Let me rephrase: it's beyond my skill to help you, and I can't reproduce it myself. However, there's a good chance that upstream (or setuptools-rust people, or PyO3 people) will be able to figure it out — it's just a matter that it would probably be more efficient if I wouldn't have to proxy between them and you ;-). (In reply to Paolo Pedroni from comment #9) > Which version of openssl do you have? I'm trying openssl-3 ATM. Could it be > the culprit? Using openssl-3 for as long as I remember ;-). Did you rebuild everything else against it? (In reply to Michał Górny from comment #11) > (In reply to Paolo Pedroni from comment #9) > > Which version of openssl do you have? I'm trying openssl-3 ATM. Could it be > > the culprit? > > Using openssl-3 for as long as I remember ;-). Did you rebuild everything > else against it? I'm quite sure I made at least an 'emerge -1e @world' since switching to openssl-3. (In reply to Michał Górny from comment #10) > Let me rephrase: it's beyond my skill to help you, and I can't reproduce it > myself. However, there's a good chance that upstream (or setuptools-rust > people, or PyO3 people) will be able to figure it out — it's just a matter > that it would probably be more efficient if I wouldn't have to proxy between > them and you ;-). It's perfectly clear. I'm working on that. There's a chance this is related to LTO. Could you rebuild the relevant packages here just to rule that out? (In reply to Sam James from comment #14) > There's a chance this is related to LTO. Could you rebuild the relevant > packages here just to rule that out? Good idea. Which packages do you mean? I tried compiling dev-lang/rust and dev-libs/openssl without LTO, without any discernible effect.
If I try compiling dev-python/cryptography itself without LTO, the error message changes and becomes:
>>> Test phase: dev-python/cryptography-40.0.1
* python3_10: running distutils-r1_run_phase python_test
python3.10 -m pytest -vv -ra -l -Wdefault --color=yes -o console_output_style=count -p no:cov -p no:flake8 -p no:flakes -p no:pylint -p no:markdown -p no:sugar -p no:xvfb -p no:tavern --ignore tests/bench -n 16
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --no-subtests-shortletter
inifile: /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1/pyproject.toml
rootdir: /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1
(same thing if I compile cryptography with clang instead of gcc)
And now:
# readelf -a /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1-python3_10/install/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so | grep PyInit
Password:
731: 00000000000e9560 526 FUNC GLOBAL DEFAULT 15 PyInit__rust
7572: 00000000002639d0 239 FUNC LOCAL DEFAULT 15 PyInit__openssl
7595: 00000000000e9560 526 FUNC GLOBAL DEFAULT 15 PyInit__rust
(In reply to Paolo Pedroni from comment #16) > __main__.py: error: unrecognized arguments: --no-subtests-shortletter > inifile: > /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1/ > pyproject.toml > rootdir: > /var/tmp/portage/dev-python/cryptography-40.0.1/work/cryptography-40.0.1 That was because I was missing dev-python/pytest-subtests. After installing it tests ran fine. The culprit was then LTO with gcc. Shall I close this bug as invalid, or put it to block "the" LTO bug? This is one of those things where I don't actually have a grasp of what the problem is, other than knowing Rust needs LTO handling specially. I think it requires thin LTO because it's all LLVM based and hence needs Clang if doing LTO. Had this problem as well, used the compiler-clang environment file in https://wiki.gentoo.org/wiki/Clang and now it works. I also ran into this. For me it was sufficient to just build dev-python/cryptography without LTO - no clang necessary (on ppc64). A filter-lto should do the trick. The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d92f1991a0f3461bf226de3e239b5588b63305b0 commit d92f1991a0f3461bf226de3e239b5588b63305b0 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-07-10 02:00:49 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-10 02:01:07 +0000 dev-python/cryptography: filter-lto Closes: https://bugs.gentoo.org/903908 Signed-off-by: Sam James <sam@gentoo.org> .../cryptography/cryptography-40.0.2-r1.ebuild | 4 + .../cryptography/cryptography-41.0.1-r1.ebuild | 149 +++++++++++++++++++++ dev-python/cryptography/cryptography-41.0.1.ebuild | 4 + 3 files changed, 157 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f24547d15dfd8e2ec62c474fea0a05d860241679 commit f24547d15dfd8e2ec62c474fea0a05d860241679 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-07-10 02:44:52 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-10 02:45:28 +0000 dev-python/cryptography: revbump for earlier filter-lto change In d92f1991a0f3461bf226de3e239b5588b63305b0, I revbumped for ~arch but not stable, but on reflection, it's not really worth the distinction and it's more likely to cause confusion given LTO is widespread nowadays. Bug: https://bugs.gentoo.org/903908 Signed-off-by: Sam James <sam@gentoo.org> ...0.2-r1.ebuild => cryptography-40.0.2-r2.ebuild} | 0 .../cryptography/cryptography-41.0.1-r1.ebuild | 2 +- dev-python/cryptography/cryptography-41.0.1.ebuild | 149 --------------------- 3 files changed, 1 insertion(+), 150 deletions(-) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=dc51935f7aae5f89d1ffecabef322680979952b8 commit dc51935f7aae5f89d1ffecabef322680979952b8 Author: Georgy Yakovlev <gyakovlev@gentoo.org> AuthorDate: 2023-02-09 00:49:47 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-12 06:24:42 +0000 cargo.eclass: filter out lto flags for C/CXX compilers we do it in src_compile to avoid excessive flag stripping in projects using cargo.eclass just to fetch crates. Bug: https://bugs.gentoo.org/903908 Closes: https://bugs.gentoo.org/893658 Closes: https://bugs.gentoo.org/910220 Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org> Signed-off-by: Sam James <sam@gentoo.org> eclass/cargo.eclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) |