While compiling python-3.13 with PGO enabled, the following test failures happen (preventing the build to complete): 0:00:11 load avg: 5.82 [130/423] test_fcntl failed (1 error) test_fcntl_64_bit (test.test_fcntl.TestFcntl.test_fcntl_64_bit) ... ERROR 0:00:27 load avg: 6.62 [272/423] test_perf_profiler failed (1 failure) test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_appear_in_the_stack_if_perf_activated) ... skipped 'Unwinding is unreliable with frame pointers' test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... skipped 'Unwinding is unreliable with frame pointers' test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated) ... FAIL test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... ok Reproducible: Always Steps to Reproduce: 1. unmask python_targets_python3_13 use flag in use.stable.mask 2. rebuild python-exec with python_targets_python3_13 use flag enabled 3. USE="pgo" emerge -av1 --autounmask =dev-lang/python-3.13.0_beta1_p3 (add to package.keywords if necessary) Actual Results: Test suite fails. Expected Results: Build and tests should succeed.
Created attachment 895410 [details] emerge --info
Created attachment 895411 [details] build.log
https://github.com/python/cpython/blob/939c201e00943c6dc2d515185168c30606ae522c/Lib/test/test_fcntl.py#L137 I found out that the fcntl failure probably stems from the fact that my linux kernel in use here doesn't have DNOTIFY. $ gzip -cd /proc/config.gz | grep DNOT # CONFIG_DNOTIFY is not set
Created attachment 895631 [details] Linux kernel config
the same happens with =dev-lang/python-3.13.0_beta2_p5.
Not really sure what to do about this. Do we really want to add a config check just for skipping a test w/ PGO? You'll get worse training data.
The second problem (test_perf_profiler) appears much more concerning to me, because I have no clue what goes wrong (gdb and perf work properly on this machine). But overall I think a solution could be to just skip the test if the necessary kernel options aren't set, or somehow downgrade these specific errors to warnings.
I can reproduce the perf failure on one machine. Will look more.
I see lots of e.g. 75cf49ffe6c5 py::foo:/var/tmp/portage/dev-lang/python-3.13.0_rc1_p3-r1/temp/ test_python_hn7x5r7r/tmpqjgfp75u/perftest.py+0x5 but no py::bar:...
The test failed on a recent CPython PR: https://github.com/python/cpython/pull/123472#issuecomment-2325419902. It's likely a kernel change or something.
I'm poking more. Is there anything interesting about your system?
enabling USE="bpf libpfm lzma unwind" on dev-util/perf seems to have helped, not sure which did it?
Okay, so USE=unwind does it, because it makes it use libunwind, not elfutils. The discussion at https://github.com/python/cpython/pull/112254#issuecomment-2096280126 onwards is insightful. But all that said: $ perf version perf version 6.10.4 ... which should have the fixes for the specific issue they mentioned. It's not clear to me why libunwind will work for me and elfutils won't, but I already spent too much time on this for now.
Unfortunately enabling USE=unwind for perf is not possible on a llvm profile because the ebuilds don't consider llvm-libunwind. [ebuild N ] sys-libs/libunwind-1.8.1 USE="zlib -debug -debug-frame -doc -libatomic -lzma -static-libs -test" ABI_X86="(32) (64) (-x32)" [ebuild R ] dev-util/perf-6.11 USE="unwind*" [blocks B ] sys-libs/libunwind ("sys-libs/libunwind" is soft blocking sys-libs/llvm-libunwind-18.1.8) * Error: The above package list contains packages which cannot be * installed at the same time on the same system.
Can you file a Gentoo bug for perf please?
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c38348609e2ee7bd933641cf32a03965fbf7a478 commit c38348609e2ee7bd933641cf32a03965fbf7a478 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2024-10-27 19:10:43 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2024-10-27 19:11:29 +0000 dev-lang/python: Skip test_perf_profiler because of frequent failures Bug: https://bugs.gentoo.org/933840 Signed-off-by: Michał Górny <mgorny@gentoo.org> dev-lang/python/python-3.13.0-r100.ebuild | 2 ++ dev-lang/python/python-3.13.0.ebuild | 2 ++ dev-lang/python/python-3.14.0_alpha1-r100.ebuild | 2 ++ dev-lang/python/python-3.14.0_alpha1.ebuild | 2 ++ 4 files changed, 8 insertions(+)
