Gentoo has recently just bumped dev-lang/python to Python 3.11. But it now includes a dependency (RDEPEND) of sys-apps/util-linux, making it unusable in a macOS Gentoo Prefix. Right now it causes bootstrap failure on macOS. These are the packages that would be merged, in order: Calculating dependencies... done! !!! All ebuilds that could satisfy "sys-apps/util-linux:=" have been masked. !!! One of the following masked packages is required to complete your request: - sys-apps/util-linux-9999::gentoo_prefix (masked by: package.mask, missing keyword) /Users/gentoo/gentoo/var/db/repos/gentoo/profiles/prefix/darwin/package.mask: # Fabian Groffen <grobian@gentoo.org> (2014-01-14) # util-linux has no business on Darwin systems - sys-apps/util-linux-2.39_rc3::gentoo_prefix (masked by: package.mask, missing keyword) - sys-apps/util-linux-2.39_rc2::gentoo_prefix (masked by: package.mask, missing keyword) - sys-apps/util-linux-2.38.1-r2::gentoo_prefix (masked by: package.mask) - sys-apps/util-linux-2.38.1::gentoo_prefix (masked by: package.mask) - sys-apps/util-linux-2.37.4::gentoo_prefix (masked by: package.mask) (dependency required by "dev-lang/python-3.11.3::gentoo_prefix" [ebuild]) (dependency required by "dev-python/inflect-6.0.4::gentoo_prefix[python_targets_python3_11]" [ebuild]) (dependency required by "dev-python/jaraco-text-3.11.1::gentoo_prefix" [ebuild]) (dependency required by "dev-python/setuptools-67.7.2::gentoo_prefix" [ebuild]) (dependency required by "dev-python/autocommand-2.2.2::gentoo_prefix" [ebuild]) For more information, see the MASKED PACKAGES section in the emerge man page or refer to the Gentoo Handbook.
From the source code, we can see that util-linux is currently a hard runtime dependency that cannot be turned off. This should be fixed. > RDEPEND=" > app-arch/bzip2:= > app-arch/xz-utils:= > app-crypt/libb2 > >=dev-libs/expat-2.1:= > dev-libs/libffi:= > dev-python/gentoo-common > sys-apps/util-linux:= > >=sys-libs/zlib-1.1.3:= > virtual/libcrypt:= > virtual/libintl > ensurepip? ( dev-python/ensurepip-wheels ) > gdbm? ( sys-libs/gdbm:=[berkdb] ) > ncurses? ( >=sys-libs/ncurses-5.2:= ) > readline? ( > !libedit? ( >=sys-libs/readline-4.1:= ) > libedit? ( dev-libs/libedit:= ) > ) > sqlite? ( >=dev-db/sqlite-3.3.8:3= ) > ssl? ( >=dev-libs/openssl-1.1.1:= ) > tk? ( > >=dev-lang/tcl-8.0:= > >=dev-lang/tk-8.0:= > dev-tcltk/blt:= > dev-tcltk/tix > ) > !!<sys-apps/sandbox-2.21 >"
Even after removing util-linux, it's still unbuilable due to a #ifdef problem in the Python source code. Python 3.11 should be masked in Gentoo Prefix (at least on macOS) for now. > ./Modules/posixmodule.c: In function 'parse_posix_spawn_flags': > ./Modules/posixmodule.c:186:64: warning: comparison between pointer and integer > 186 | (posix_spawn != NULL && setsid != NULL) > | ^~ > ./Modules/posixmodule.c:6026:13: note: in expansion of macro 'HAVE_POSIX_SPAWN_SETSID_RUNTIME' > 6026 | if (HAVE_POSIX_SPAWN_SETSID_RUNTIME) { > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > ./Modules/posixmodule.c: In function 'os_mkfifo_impl': > ./Modules/posixmodule.c:10690:17: error: 'HAVE_MKFIFOAT_RUNTIME' undeclared (first use in this function); did you mean 'HAVE_MKDIRAT_RUNTIME'? > 10690 | if (HAVE_MKFIFOAT_RUNTIME) { > | ^~~~~~~~~~~~~~~~~~~~~ > | HAVE_MKDIRAT_RUNTIME > ./Modules/posixmodule.c:10690:17: note: each undeclared identifier is reported only once for each function it appears in > ./Modules/posixmodule.c: In function 'os_mknod_impl': > ./Modules/posixmodule.c:10759:17: error: 'HAVE_MKNODAT_RUNTIME' undeclared (first use in this function); did you mean 'HAVE_MKDIRAT_RUNTIME'? > 10759 | if (HAVE_MKNODAT_RUNTIME) { > | ^~~~~~~~~~~~~~~~~~~~ > | HAVE_MKDIRAT_RUNTIME > ./Modules/posixmodule.c: In function 'probe_mkfifoat': > arm64-apple-darwin22-gcc -Wsign-compare -DNDEBUG -O2 -pipe -fwrapv -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -I/Users/gentoo/gentoo/tmp/usr/include/ncursesw -DPy_BUILD_CORE_BUILTIN -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o > ./Modules/posixmodule.c:15647:23: error: 'HAVE_MKFIFOAT_RUNTIME' undeclared (first use in this function); did you mean 'HAVE_MKDIRAT_RUNTIME'? > 15647 | PROBE(probe_mkfifoat, HAVE_MKFIFOAT_RUNTIME) > | ^~~~~~~~~~~~~~~~~~~~~ > ./Modules/posixmodule.c:15611:11: note: in definition of macro 'PROBE' > 15611 | if (test) { \ > | ^~~~ > ./Modules/posixmodule.c: In function 'probe_mknodat': > ./Modules/posixmodule.c:15651:22: error: 'HAVE_MKNODAT_RUNTIME' undeclared (first use in this function); did you mean 'HAVE_MKDIRAT_RUNTIME'? > 15651 | PROBE(probe_mknodat, HAVE_MKNODAT_RUNTIME) > | ^~~~~~~~~~~~~~~~~~~~ > ./Modules/posixmodule.c:15611:11: note: in definition of macro 'PROBE' > 15611 | if (test) { \ > | ^~~~ > make: *** [Makefile:2695: Modules/posixmodule.o] Error 1 > make: *** Waiting for unfinished jobs.... > * ERROR: dev-lang/python-3.11.3::gentoo_prefix failed (compile phase): > * emake failed By inspecting the Python source code /Modules/posixmodule.c, it seems that Python conditionally defines HAVE_MKFIFOAT_RUNTIME and HAVE_MKDIRAT_RUNTIME based on "#ifdef" checks. It seems that the the Clang specific "__builtin_available()" compiler built-in function is used to check them, but since we're using GCC, we fail the HAVE_BUILTIN_AVAILABLE checks and these macros are never defined. The "else" clause of the "#ifdef" has fallback definitions in case "__builtin_available()" is not available. But when the new HAVE_MKFIFOAT_RUNTIME and HAVE_MKDIRAT_RUNTIM flags were added into Python, no fallback definitions were provided, thus it previously worked but now fails. > #ifdef HAVE_BUILTIN_AVAILABLE > [...] > # define HAVE_MKFIFOAT_RUNTIME __builtin_available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *) > # define HAVE_MKNODAT_RUNTIME __builtin_available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *) > # define HAVE_POSIX_SPAWN_SETSID_RUNTIME __builtin_available(macOS 10.15, *) > #else /* Xcode 8 or earlier */
I reported the problem to the upstream: https://github.com/python/cpython/issues/104106
Two changes are needed to mask Python 3.11. First, Python 3.11 should be masked on Darwin. Also, the PYTHON_TARGETS="python3_10" should be the default make.conf option on Darwin for now, so that Python packages such as Portage can be installed.
Wait a minute, I found a solution to address the root clause. Perhaps we don't need to mask Python 3.11... From the git log I found the regression was introduced upstream in 6d0a0191 ("Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK"). > The macOS 13 SDK includes support for the `mkfifoat` and `mknodat` system calls. > Using the `dir_fd` option with either `os.mkfifo` or `os.mknod` could result in a > segfault if cpython is built with the macOS 13 SDK but run on an earlier > version of macOS. Prevent this by adding runtime support for detection of > these system calls ("weaklinking") as is done for other newer syscalls on > macOS. Since moving a Gentoo Prefix to a different host OS version is never supported, we don't need this fix. So the temporary fix is simply to patch the code and revert this commit.
I've committed a fix into my private tree, awaiting for further testing.
I've just tested the patch and confirmed that it's working, I just opened a Pull Request at https://github.com/gentoo/prefix/pull/29.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=dd1bf7964984cd244b9ccc5f002fe2cafb0f3ded commit dd1bf7964984cd244b9ccc5f002fe2cafb0f3ded Author: Sam James <sam@gentoo.org> AuthorDate: 2023-05-03 06:23:20 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-05-03 06:23:21 +0000 dev-lang/python: conditionalise util-linux dep We don't use ::gentoo python on macOS prefix yet but this is worth doing anyway given it was wrong before & it makes the diff smaller. Bug: https://bugs.gentoo.org/905618 Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.10_p3.ebuild | 2 +- dev-lang/python/python-3.10.11.ebuild | 2 +- dev-lang/python/python-3.11.2_p2.ebuild | 2 +- dev-lang/python/python-3.11.3.ebuild | 2 +- dev-lang/python/python-3.12.0_alpha6.ebuild | 2 +- dev-lang/python/python-3.12.0_alpha7.ebuild | 2 +- dev-lang/python/python-3.8.16_p4.ebuild | 2 +- dev-lang/python/python-3.9.16_p3-r1.ebuild | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-)
util-linux dep was made conditional again for Prefix in https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=a4438165828305afede8aa1d1e2e76fa9ea5a09e
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=f3ecbf1a719175f27f305bb4d53a1f6c5c32a041 commit f3ecbf1a719175f27f305bb4d53a1f6c5c32a041 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2023-05-04 07:33:30 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2023-05-04 07:33:30 +0000 dev-lang/python-3.11.3: bump patchset for Ventura GCC patch Bug: https://bugs.gentoo.org/905618 Signed-off-by: Fabian Groffen <grobian@gentoo.org> dev-lang/python/python-3.11.3.ebuild | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Hold up. Waite a minute. Has this been fixed already? I tried to update my system this afternoon (GMT time) and the issue still persists.
(In reply to Askar Bektassov from comment #11) > Hold up. Waite a minute. Has this been fixed already? I tried to update my > system this afternoon (GMT time) and the issue still persists. No, it's not fixed, if you read the bug report you would see "Status: CONFIRMED", rather than "RESOLVED FIXED". Just because a single patch has been committed doesn't imply it has actually been fixed. It's still a work-in-progress and requires multiple commits to fix. The patch still needs another round of rework and review.
I've just updated my Pull Request which includes the official upstream patch.
(In reply to Tom Li from comment #12) > (In reply to Askar Bektassov from comment #11) > > Hold up. Waite a minute. Has this been fixed already? I tried to update my > > system this afternoon (GMT time) and the issue still persists. > > No, it's not fixed, if you read the bug report you would see "Status: > CONFIRMED", rather than "RESOLVED FIXED". Correction: as of commit f3ecbf1a719175f27f305bb4d53a1f6c5c32a041 [1], the patch has already been included into the Python 3.11 Gentoo Prefix Patchset r1. But it may need another day or two for the mirrors to catch up. At the time of writing, the rsync mirror is now up-to-date, but the Web snapshot is not updated yet. To confirm whether the patch has been synced into your system, please check whether the Python ebuild says: PREFIX_PATCHSET="python-prefix-gentoo-${MY_PV}-patches-r1" [1] https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=f3ecbf1a719175f27f305bb4d53a1f6c5c32a041
Unfortunately it has not been fixed. Fabian Groffen accidentally included the previously broken version of the upstream patch into the Gentoo Prefix patchset, which contained a typo so it doesn't work. +# ifdef HAVE_MKNODAT_RUNTIME +# define HAVE_MKNODAT_RUNTIME (mknodat != NULL) +# endif "ifdef HAVE_MKNODAT_RUNTIME" is wrong and doesn't do anything, it should've been "ifdef HAVE_MKNODAT" Please tell Fabian Groffen to update the patch to the fixed version.
(In reply to Tom Li from comment #15) > Unfortunately it has not been fixed. Fabian Groffen accidentally included > the previously broken version of the upstream patch into the Gentoo Prefix > patchset, which contained a typo so it doesn't work. > > +# ifdef HAVE_MKNODAT_RUNTIME > +# define HAVE_MKNODAT_RUNTIME (mknodat != NULL) > +# endif > > "ifdef HAVE_MKNODAT_RUNTIME" is wrong and doesn't do anything, it should've > been "ifdef HAVE_MKNODAT" > > Please tell Fabian Groffen to update the patch to the fixed version. The broken patch was committed into the Gentoo Prefix tree at 2023-05-04 07:33:30 UTC, meanwhile I already pointed out that the upstream patch is broken to the upstream Python developers at 2023-05-03 22:44:34, which was then fixed at 2023-05-04 01:17:25 UTC. It should've been entirely preventable.
Sure, or he did the work the night before and pushed it when he woke up. I appreciate the help (a lot), but please can you be a little bit less testy?
(In reply to Sam James from comment #17) > Sure, or he did the work the night before and pushed it when he woke up. I > appreciate the help (a lot), but please can you be a little bit less testy? As everyone knows, most communities work asynchronously, a simple change can take a long time to land, and on top of that, build releases and mirrors create additional delay. As a result, there's a significant delay between the moment a fix is developed and the moment downstream users receive the same fix. Thus, it's important to maximize the efficient use of time, so that downstream users receive a fix as soon as possible (to the extend allowed by maintainers personal time). And it's especially to avoid unproductive work, which serves no purpose but wastes everyone's time, including project maintainers themselves' already limited time. To put that in perspective. Two days ago, I submitted a 100% tested patch that would fix this problem. Unfortunately, it was not merged at that time because the maintainer apparently didn't believe the existence of the bug. To ensure that the bug was not my imagination, I then spend much time on rebuilding GCC 12.1 and GCC 12.2, and testing both upstream source code and downstream Python ebuild, this reaffirmed that the problem could be replicated reliably. I also pointed out that the problem has already been acknowledged by the upstream. It's only at this point that I managed to persuade the maintainer that the bug is real, and no action has been taken during the first day. And on the second day, a broken patch was pushed into the tree without being noticed, understandingly, as Groffen apparently lacked the required system to replicate this issue. It was not noticed before the end of the second time. But If the original pull request was merged, it could've saved everyone two days of time. It was a workaround that simply reverts the regression rather than fixes it cleanly for sure, but the end result would not be that different, since both patches bypassed the clang's runtime feature check when GCC is used.
In many other projects like the Linux Kernel Mailing List, if a patch is submitted, by convention, changing the patch is the responsibility of the submitter, almost never the maintainers. A maintainer would usually take no action to modify the patch further, and the problem is always fixed by at the source by the submitter. I used to think that it's a waste of time, as sometimes even a small coding style problem like a single trailing whitespace can block the entire patch from being accepted. If the fix is "obvious", why can't it be fixed by maintainers right away? But my recent attempts on trying to submit patches to Gentoo have unfortunately caused two incorrect patches to be committed into the tree, on two separate cases. One for a PIE flag fix for GCC 12.2, and this fix for Python 11. Both mistakes were appearently caused by system environment differences and the subsequent misunderstandings from it. Ordinarily, I would have expected that a project maintainer would either keep a patch stand-still, or bring a patch forward. But I've never experienced a case when the maintainer would bring a patch "backward" as a result of this kind of miscommunications. Now it makes me rethink about the policy of "maintainers don't help people to change patches", under this traditional policy, the problem of committing broken patches would've been totally eliminated. Nevetheless, ultimately, it's the responsibility of contributors to understand the mode of operation of a project and acts accordingly. I incorrectly assumed Gentoo is operating in the same way as other projects like LKML. On this basis, I apologize for any inconvenience it has caused. In the future, I would adjust my behavior accordingly to reduce the chance of miscommunication, for example, by not posting any comment or patch in the bug tracker until I'm totally sure about it to prevent similar premature changes from occuring in the future.
Just some background from me here. First off I think I've contributed to a tremendous fsck-up here, which is not what I'm proud of. Python and the toolchain have been challenging to Gentoo Prefix over the years, and that is the reason why I don't really pro-actively update them. Admittedly Python has become much less porting work over the years, most patches apply, but still it is a beast that always breaks. Now you only saw macOS, but also on Solaris/OpenIndiana it breaks/doesn't work at all. Add to the mix that Gentoo main decided to enable Python 3.11 a month or 2 earlier than originally announced, and I have to make a last-minute port of Python 3.11, which normally would be in masked mode introduced, so I can properly run it for a while before unleashing it. That couldn't happen now, because I knew changing the default back to older Python 3.10 would only result in other problems as versions are rapidly replacing themselves here. As it stands today, Python 3.11 does function somewhat on some of my machines/setups, but perhaps not on the most interesting one Ventura on SI. That's a shame. It really is. But please be aware that SI and latest macOS are broken for a long time, unfortunately. Contributions there are desperately needed. Yours, therefore are much appreciated, even though you may feel at this point they are not. Regarding GCC, 12.2 was for a bit enabled, then masked because it gave me issues and I didn't have time to figure them out. You've done that work, and so we could enable it, and I've tried to do what had to be done -- blind -- based on your instructions. Again, I may have failed there, my apologies for that. I was just trying to help with the cycles and (commit) access I have. Finally, I'm very interested in getting the darwin-xtools running on arm64, for which you seem to have done the work. I just don't want to commit that one blindly, because I would like to really test that myself, and yes that means I should build a proper 13-based Prefix somewhere, first with the current setup, then with the darwin-xtools variant. I expected it to be a nice-to-have rather than a must-have, so I prioritised it lower than fixing the imminent issues of gcc-12.2 (necessary to fix compiler/linkage problems e.g. with Vim) and updating to Python-3.11 to follow main. I hope this gives you a little insight in our doings and that it makes it clear that there is no ill-intent here. Your contributions are very highly appreciated. Thanks, Fabian
I believe this is now fixed in the Gentoo Prefix tree
Works perfectly on my M1 with 13.3.1(a). Thanks to both of you for your hard work on these very tricky problems, which I assume has made you millionaires by now.
