Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 886491 - bootstrap-prefix.sh fails with macOS 13 (Ventura)
Summary: bootstrap-prefix.sh fails with macOS 13 (Ventura)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: All OS X
: Normal blocker with 2 votes (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords:
: 879897 (view as bug list)
Depends on: 871324 895330 895332 895334 895336 895494 896330 898610 904474 904887 905131 905152 905618 905619
Blocks:
  Show dependency tree
 
Reported: 2022-12-17 20:26 UTC by *
Modified: 2023-06-06 20:17 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
gcc-build-logs.tar.bz2 (gcc-build-logs.tar.bz2,578.76 KB, application/x-bzip2)
2022-12-17 20:27 UTC, *
Details
gcc-12.2.0 build.log (build.log.xz,351.04 KB, application/x-xz)
2022-12-22 23:51 UTC, Yuan Liao (Leo3418)
Details
gcc-12.2.0 gcc-build-logs.tar.bz2 (gcc-build-logs.tar.bz2,599.67 KB, application/x-bzip2)
2022-12-22 23:52 UTC, Yuan Liao (Leo3418)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description * 2022-12-17 20:26:35 UTC
It makes it all the way through stage1, then fails with gcc-12.1.0 in stage2:

make[2]: Entering directory '/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/gcc'
/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/./gcc/xgcc -B/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/./gcc/ -xc++ -nostdinc /dev/null -S -o /dev/null -fself-test=/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/gcc-12-branch-gcc-12.1-darwin-r0/gcc/testsuite/selftests
<built-in>: error: unknown value '13.0' of '-mmacosx-version-min'

I'll attach the log.

The above seems identical to Bug 879897 despite this being 13.1, not 13.0

Reproducible: Always

Steps to Reproduce:
Run bootstrap-prefix.sh.
Comment 1 * 2022-12-17 20:27:27 UTC
Created attachment 843243 [details]
gcc-build-logs.tar.bz2
Comment 2 Yuan Liao (Leo3418) 2022-12-22 23:48:05 UTC
I did some research on this bug and it seems to be related to GCC 12.1 only.

I first came across <https://github.com/Homebrew/discussions/discussions/3832>, where the same error message is mentioned, and one reply there indicates that it should be fixed in GCC 12.2.

Then, I tried to build sys-devel/gcc-12.2.0::gentoo_prefix (of course ignoring the comment in that ebuild saying it would fail to compile), and I was able to get the build a little bit past the self-tests:

make[2]: Entering directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc
-12.2.0/work/build/gcc'
/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/./gcc/xgcc -B/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/./gcc/ -xc -nostdinc /dev/null -S -o /dev/null -fself-test=/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/gcc-12-branch-gcc-12.2-darwin-r0/gcc/testsuite/selftests
cc1: note: self-tests are not enabled in this build
make[2]: Leaving directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
make[2]: Entering directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
echo timestamp > s-selftest-c
make[2]: Leaving directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
make[2]: Entering directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/./gcc/xgcc -B/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/./gcc/ -xc++ -nostdinc /dev/null -S -o /dev/null -fself-test=/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/gcc-12-branch-gcc-12.2-darwin-r0/gcc/testsuite/selftests
cc1plus: note: self-tests are not enabled in this build
make[2]: Leaving directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
make[2]: Entering directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'
echo timestamp > s-selftest-c++
make[2]: Leaving directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/gcc'

But only after a while, a different error popped up:

make[3]: Entering directory '/Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/build/libcc1'
/Users/leo/Gentoo/tmp/bin/bash ./libtool --tag=CXX   --mode=link g++ -m64 -std=gnu++11 -W -Wall  -fvisibility=hidden  -Wl,-undefined,dynamic_lookup -I/Users/leo/Gentoo/tmp/usr/include -pipe -O2 -module -export-symbols /Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/gcc-12-branch-gcc-12.2-darwin-r0/libcc1/libcc1.sym -Wc,-nodefaultrpaths,-nodefaultexport -Wl,-rpath,@loader_path  '-Wl,-search_paths_first' '-L/Users/leo/Gentoo/tmp/usr/lib' -o libcc1.la -rpath /Users/leo/Gentoo/tmp/usr/lib/ findcomp.lo libcc1.lo libcp1.lo compiler.lo names.lo callbacks.lo connection.lo marshall.lo    -Wc,../libiberty/pic/libiberty.a 
libtool: link: sed -e 's,^,_,' < /Users/leo/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.2.0/work/gcc-12-branch-gcc-12.2-darwin-r0/libcc1/libcc1.sym > .libs/libcc1-symbols.expsym
libtool: link: g++ -m64 -std=gnu++11  -o .libs/libcc1.0.so -bundle  .libs/findcomp.o .libs/libcc1.o .libs/libcp1.o .libs/compiler.o .libs/names.o .libs/callbacks.o .libs/connection.o .libs/marshall.o   -L/Users/leo/Gentoo/tmp/usr/lib  -m64 -Wl,-undefined -Wl,dynamic_lookup -nodefaultrpaths -nodefaultexport -Wl,-rpath -Wl,@loader_path -Wl,-search_paths_first ../libiberty/pic/libiberty.a   -Wl,-exported_symbols_list,.libs/libcc1-symbols.expsym
clang: error: unknown argument: '-nodefaultrpaths'
clang: error: unknown argument: '-nodefaultexport'
make[3]: *** [Makefile:576: libcc1.la] Error 1

I will upload build logs for GCC 12.2.  Please let me know if I shall open another bug for 12.2.
Comment 3 Yuan Liao (Leo3418) 2022-12-22 23:51:48 UTC
Created attachment 844785 [details]
gcc-12.2.0 build.log
Comment 4 Yuan Liao (Leo3418) 2022-12-22 23:52:49 UTC
Created attachment 844787 [details]
gcc-12.2.0 gcc-build-logs.tar.bz2
Comment 5 * 2023-01-29 23:55:03 UTC
*** Bug 879897 has been marked as a duplicate of this bug. ***
Comment 6 * 2023-01-29 23:59:32 UTC
Same error in macOS 13.2:

make[2]: Entering directory '/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/gcc'
/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/./gcc/xgcc -B/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/build/./gcc/ -xc++ -nostdinc /dev/null -S -o /dev/null -fself-test=/opt/gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/gcc-12-branch-gcc-12.1-darwin-r0/gcc/testsuite/selftests
<built-in>: error: unknown value '13.0' of '-mmacosx-version-min'


I guess we need a newer version of gcc?
Comment 7 Alexey 2023-02-01 19:46:00 UTC
I successfully got past this stage by using gcc-12 installed from homebrew.

$ brew install gcc
$ cd ~/Gentoo/tmp/bin
$ ln -s /opt/homebrew/bin/gcc-12 .
$ ln -s /opt/homebrew/bin/g++-12 .

Then update this in bootstrap-prefix.sh near line 194:

-CC=gcc
-CXX=g++
+CC=gcc-12
+CXX=g++-12

Because the code below still checks version of /usr/bin/gcc instead of $CC, I think it still follows branch of linker=sys-devel/native-cctools and isgcc="".

sys-devel/gcc-12.2.0 succeeded after this.

I don't know what brew does differently to make gcc work. Or whether it would be a good idea or not to update the bootstrap-prefix.sh script to take gcc from brew for the bootstrap.

---

Darwin Kernel Version 22.3.0: Thu Jan  5 20:48:54 PST 2023; root:xnu-8792.81.2~2/RELEASE_ARM64_T6000 arm64

$ gcc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ gcc-12 --version
gcc-12 (Homebrew GCC 12.2.0) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-01 20:25:25 UTC
After chatting w/ DarthGandalf in -prefix: I went ahead and added ~arm64-macos keywords to everything but gcc in the overlay, as it's annoying if the script stops due to an issue that isn't a build failure, and all of them should work modulo possible autoconf issues, and we'd rather get a build.log if those fail.
Comment 9 Tom Li 2023-02-18 05:28:59 UTC
Both GCC 12.1 and GCC 12.2 have problems, but to me I found the GCC 12.1 problems (the version used by default) were easier to solve. GCC 12.2 has its own problems and I haven't figured out how to solve them.

* Darwin: Future-proof -mmacosx-version-min
> f18cbc1ee1f4 (2021-12-18) updated various parts of gcc to not impose a
> Darwin or macOS version maximum of the current known release. [...] However, 
> f18cbc1ee1f4 missed config/darwin-c.c (now .cc), which continued to impose a 
> maximum of macOS 12 on the -mmacosx-version-min compiler driver argument.

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=6725f186cb70d48338f69456864bf469a12ee5be

* libstdc++: Rename __null_terminated to avoid collision with Apple SDK
> The macOS 13 SDK (and equivalent-version iOS and other Apple OS SDKs)
> contain this definition in <sys/cdefs.h>: "#define __null_terminated" This
> collides with the use of __null_terminated in libstdc++'s experimental
> fs_path.h.

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=d1201dbf55a11d391030914985ba6b443e59baa5

The second patch has landed in GCC 12.2, but the first patch did not (if I recall correctly), so all GCC 12 versions need this patch.

After modifying the ebuild and applying both patches, GCC has successfully finished Gentoo Stage3 bootstrap.

Unfortunately, Stage3 bootstrap failed to progress beyond GCC, it later failed to find Python dependencies needed by Portage due to the old tree. Manually adding Python 3.9 keywords and unmasking packages made it run, until it failed in the mid-way again.

During the merge of app-text/xmlto-0.0.28-r9, flex failed to launch GNU m4 due to an internal fatal error, I haven't find a solution yet:

make[1]: Entering directory '/Users/apple/gentoo/var/tmp/portage/app-text/xmlto-0.0.28-r9/work/xmlto-0.0.28'
/Users/apple/Gentoo/tmp/bin/bash ./ylwrap xmlif/xmlif.l unknown.c xmlif/xmlif.c -- /Users/apple/Gentoo/tmp/bin/bash '/Users/apple/Gentoo/var/tmp/portage/app-t
ext/xmlto-0.0.28-r9/work/xmlto-0.0.28/missing' flex
flex: fatal internal error, exec of /Users/apple/Gentoo/usr/bin/gm4 failed                                       
flex: error writing output file lex.yy.c                                                                         
make[1]: *** [Makefile:777: xmlif/xmlif.c] Error 1

But both flex and GNU m4 themselves could be emerged without problems, I don't understand why.

The symptom is similar to a macOS bug reported to the upstream, currently no developer has responded it:

https://github.com/westes/flex/issues/539
Comment 10 Tom Li 2023-02-18 20:57:13 UTC
I've just successfully completed the bootstrap.

There are mainly three problems:

1. GCC 12.1 is incompatible with macOS 13.2 (Ventura) and needs two patches, as I mentioned before.

2. During stage3 bootstrap, GNU gettext fails with Undefined symbols for architecture arm64: "_gl_get_setlocale_null_lock" because libintl was not present during the early stage (though it would be installed later). This is a known bug (GNU gettext bug #62659 [1]), and currently the Portage tree already contain a patch for musl [2].

use elibc_musl && eapply "${FILESDIR}"/${PN}-0.21-musl-omit_setlocale_lock.patch

It just needs to be applied to Darwin as well, as in

use elibc_Darwin && eapply "${FILESDIR}"/${PN}-0.21-musl-omit_setlocale_lock.patch

3. Missing keywords for various Python dependencies for portage, including dev-python/jaraco-text and others.

The m4 problem I previous discussed was probably just a side-effect due to a non-clean system, it was not replicated in a fresh install.

[1] https://savannah.gnu.org/bugs/?62659

[2] https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-devel/gettext/gettext-0.21.1.ebuild
Comment 11 Tom Li 2023-02-19 17:49:50 UTC
Significant progress has been made on investigating the mechanism behind the GCC 12.2 problem and fixing the GCC 12.1 problem.

See: 

https://bugs.gentoo.org/895332 (GCC 12.1)
https://bugs.gentoo.org/895334 (GCC 12.2)

Now I'm confident that both problems can be easily fixed.
Comment 12 Tom Li 2023-02-19 19:06:38 UTC
The fix for GCC 12.1.0 has been merged into the upstream, GCC 12.2.0 is also masked until a fix could be made. Now Bug 895332 is closed, the GCC bootstrap should work without problems. Furthermore, Bug 895336 is closed, so the Python dependencies needed by portage have all been added.

The remaining problem is GNU gettext in Bug 895330. Finally, a real fix of GCC 12.2.0 in Bug 895334 is still needed. But now its nature has been investigated with success, fixing it should be easy.

For now, if you want to bootstrap Gentoo Prefix on macOS 13 with Apple M1, you can follow this procedure:

1. export LATEST_TREE_YES=1
Unlike what's suggested in Gentoo Wiki, you must set this variable before the beginning of stage1.

2. ${EPREFIX}/bootstrap-prefix.sh ${EPREFIX} stage1
This should proceed without any problem.

3. ${EPREFIX}/bootstrap-prefix.sh ${EPREFIX} stage2
This should proceed without any problem.

4. Apply patch for Bug 895330.
https://github.com/gentoo/gentoo/pull/29655

5. ${EPREFIX}/bootstrap-prefix.sh ${EPREFIX} stage3

See https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap for more information.
Comment 13 Tom Li 2023-02-19 19:51:57 UTC
The patch for GNU gettext has also been merged, step 4 is no longer necessary. Theoretically, the bootstrap is now working with LATEST_TREE_YES. It may take a few hours before the changes are picked up by all mirrors.
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-19 19:58:36 UTC
(In reply to Tom Li from comment #12)
> 1. export LATEST_TREE_YES=1
> Unlike what's suggested in Gentoo Wiki, you must set this variable before
> the beginning of stage1.

Fixed: https://wiki.gentoo.org/index.php?title=Project:Prefix/Manual_Bootstrap&diff=1186134&oldid=635276
Comment 15 Tom Li 2023-02-20 03:57:49 UTC
The previously mentioned "flex" problem was not a coincidence. I just realized that the problem still exists, it just wasn't being triggered during the bootstrap with the latest tree, but it may come back in the future.

ec2-user@ip-10-0-4-85 ~ $ flex
<stdin>:1: premature EOF
flex: fatal internal error, exec of /Users/ec2-user/gentoo/usr/bin/gm4 failed

This is the next target to investigate.
Comment 16 Tom Li 2023-02-20 23:08:38 UTC
The flex bug has been fixed. Now macOS bootstrap should work flawlessly. Also, GCC 12.2.0 can be manually unmasked and installed inside the prefix once the bootstrap is successful using GCC 12.1.0 by default.

But the GCC 12.2 bug still needs a permanent fix. The easiest option is to simply pass "--disable-darwin-at-rpath" to GCC on macOS, this disables the new rpath feature during early bootstrap, allowing clang to link GCC objects. However, it means all GCC can only output binaries with hardcoded rpath. Although this is safe on Gentoo prefix, because moving the prefix is unsupported anyway. But this would be major restriction to end-users who wishes to use Gentoo's GCC to build projects outside Gentoo Prefix.

I'm no ebuild guru. Does anyone know if there's a way to run code in an ebuild file at, during and ONLY during the process of Gentoo Prefix bootstrap (not GCC bootstrap). Please advice.

Failing that, I think I'll just hack it to run if the GCC "bootstrap" keyword is unset. The three-stage GCC bootstrap is only normally skipped during the three-stage Gentoo bootstrap, and it's not a recommended setting for end users, so the risk of not supporting dynamic rpath is small.
Comment 17 Fabian Groffen gentoo-dev 2023-02-21 07:34:32 UTC
thanks for your work!

There is a "bootstrap" USE-flag that used to be set during bootstrap, and not at the final emerge -e @world stage.  That would allow you to just disable the rpath feature when USE=bootstrap is set, perhaps something like
 EXTRA_ECONF+=" $(use_enable !bootstrap darwin-at-rpath)"
for the darwin case in src_configure.
Comment 18 * 2023-02-23 17:23:22 UTC
Thanks for your work. Not sure if anything changed in the last 24 hours, but it's failing for me in stage2:

# LATEST_TREE_YES=1 bootstrap-prefix.sh

...

* prefix-portage-3.0.30.1 successfully bootstrapped
* stage1 successfully finished
* Bootstrapping Gentoo prefixed portage installation using
* host:   arm64-apple-darwin22
* prefix: /opt/gentoo
* ready to bootstrap stage2_log
* Triggering Darwin with GCC toolchain
USE=-acl -berkdb -fortran -gdbm -git -libcxx -nls -pcre -python -qmanifest -qtegrity -readline -sanitize bootstrap clang internal-glib PKG=sys-devel/gnuconfig

Performing Global Updates

,,,

!!! BINPKG_COMPRESS unsupported zstd. Missing package: app-arch/zstd
These are the packages that would be merged, in order:


!!! All ebuilds that could satisfy "sys-devel/gnuconfig" have been masked.
!!! One of the following masked packages is required to complete your request:
- sys-devel/gnuconfig-99999999::gentoo_prefix (masked by: missing keyword)
- sys-devel/gnuconfig-20221007::gentoo_prefix (masked by: missing keyword)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.
Comment 19 Tom Li 2023-02-24 03:15:10 UTC
(In reply to * from comment #18)
> Thanks for your work. Not sure if anything changed in the last 24 hours, but
> it's failing for me in stage2:
> 
> # LATEST_TREE_YES=1 bootstrap-prefix.sh
> 
> ...
> !!! All ebuilds that could satisfy "sys-devel/gnuconfig" have been masked.
> !!! One of the following masked packages is required to complete your
> request:
> - sys-devel/gnuconfig-99999999::gentoo_prefix (masked by: missing keyword)
> - sys-devel/gnuconfig-20221007::gentoo_prefix (masked by: missing keyword)
> 
> For more information, see the MASKED PACKAGES section in the emerge
> man page or refer to the Gentoo Handbook.

In the past, amd64 keywords were also used as arm64 keywords for macOS prefix, so packages with existing amd64 keywords continued to work on the Apple M1 ARM chip. However, after I reported several problems that only exist on ARM, Gentoo developers made some changes, and unfortunately now ARM is no longer given the same treatment as arm64, so it creates a new batch of keyword problems.

It means someone must redo a manual bootstrap and add all the missing keywords manually again.
Comment 20 Yuan Liao (Leo3418) 2023-02-24 03:46:13 UTC
(In reply to Tom Li from comment #19)
> It means someone must redo a manual bootstrap and add all the missing
> keywords manually again.
I'm glad to report that I have recently successfully bootstrapped Prefix with the '~x64-macos' keyword on Apple M1.

(In reply to * from comment #18)
> Thanks for your work. Not sure if anything changed in the last 24 hours, but
> it's failing for me in stage2:
> !!! All ebuilds that could satisfy "sys-devel/gnuconfig" have been masked.
> !!! One of the following masked packages is required to complete your
> request:
> - sys-devel/gnuconfig-99999999::gentoo_prefix (masked by: missing keyword)
> - sys-devel/gnuconfig-20221007::gentoo_prefix (masked by: missing keyword)
> 
> For more information, see the MASKED PACKAGES section in the emerge
> man page or refer to the Gentoo Handbook.
Add this line to ${EPREFIX}/etc/portage/make.conf:

    ACCEPT_KEYWORDS="${ACCEPT_KEYWORDS} ~x64-macos"
Comment 21 Fabian Groffen gentoo-dev 2023-02-24 08:26:41 UTC
Indeed, manually setting ACCEPT_KEYWORDS to "~x64-macos" brings your prefix back to the state before the sync.
Comment 22 * 2023-02-24 18:23:26 UTC
Thanks, got farther with that:
LATEST_TREE_YES=1 ACCEPT_KEYWORDS="~x64-macos" bootstrap-prefix.sh

But now it's failing during stage3 when building bash. Created Bug 896330.
Comment 23 * 2023-03-06 00:20:42 UTC
Not sure what changed, but this is currently working:
LATEST_TREE_YES=1 ACCEPT_KEYWORDS="~x64-macos" bootstrap-prefix.sh

After that finished, I added this to my make.conf:
ACCEPT_KEYWORDS="~x64-macos"

And this to my package.accept_keywords:
=sys-devel/gcc-12.2.0:12::gentoo_prefix ~*

And I was able to build gcc-12.2 (then emerge -e @world).

Thanks!
Comment 24 * 2023-04-03 16:37:23 UTC
I had to bootstrap again after upgrading to 13.3 and the above still works.
Comment 25 Tom Li 2023-04-22 11:52:32 UTC
(In reply to Fabian Groffen from comment #21)
> Indeed, manually setting ACCEPT_KEYWORDS to "~x64-macos" brings your prefix
> back to the state before the sync.

I just took a look and I found more than 100 packages need to be manually unmasked to allow even a stage3 prefix bootstrap. Therefore, I'm seriously dissatisfied with the decision to split "~arm64-macos" from "~x64-macos".

What problem is it supposed it solve?

If it's meant to prevent incompatible packages from being installed on the system,  the collateral damage it produced is tremendous! Right now, it effectively prevent entirety of Gentoo from being installed, as none of the Gentoo package so far is marked with the keyword ~amd64-macos, not even sys-apps/baselayout! One needs a massive commit to tag at least 100 with the keyword ~arm64-macos. As a result, one needs a massive search-and-replace effort in the Portage tree, and that would take us back to the point we started with. In my opinion, it doesn't solve any real problem.
Comment 26 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-04-22 11:54:24 UTC
(In reply to Tom Li from comment #25)
> (In reply to Fabian Groffen from comment #21)
> > Indeed, manually setting ACCEPT_KEYWORDS to "~x64-macos" brings your prefix
> > back to the state before the sync.
> 
> I just took a look and I found more than 100 packages need to be manually
> unmasked to allow even a stage3 prefix bootstrap. Therefore, I'm seriously
> dissatisfied with the decision to split "~arm64-macos" from "~x64-macos".

OK.

> 
> What problem is it supposed it solve?


Marking something as working when it definitely doesn't? That's what our keywording system is for.

> 
> If it's meant to prevent incompatible packages from being installed on the
> system,  the collateral damage it produced is tremendous! Right now, it
> effectively prevent entirety of Gentoo from being installed, as none of the
> Gentoo package so far is marked with the keyword ~amd64-macos, not even
> sys-apps/baselayout! One needs a massive commit to tag at least 100 with the
> keyword ~arm64-macos. As a result, one needs a massive search-and-replace
> effort in the Portage tree, and that would take us back to the point we
> started with. In my opinion, it doesn't solve any real problem.

It solves the problem of things like b2 and flex being tagged as "working" when they didn't.

Anyway, see bug 904474. I can commit all of those if they work from your memory.
Comment 27 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-04-22 11:56:19 UTC
(Ultimately, the whole point of keywording is to allow visiblility checks so you can have depgraph consistency.)
Comment 28 Tom Li 2023-04-22 12:06:19 UTC
> It solves the problem of things like b2 and flex being tagged as "working" when they didn't.

Let's continue the discussion at #904474.
Comment 29 Tom Li 2023-04-23 15:31:29 UTC
Please change the title from "bootstrap-prefix.sh fails in stage2 with macOS 13.2 (Ventura)" to "bootstrap-prefix.sh fails with macOS 13.2 (Ventura)", removing the words "stage2". There are multiple problems that sometimes affect stage2 and sometimes affect stage3, it would be the best if these issues can be tracked together here.
Comment 30 * 2023-04-23 18:37:07 UTC
Could we just say Ventura? I assume most people are on 13.3 by now.
Comment 31 Tom Li 2023-04-24 15:05:21 UTC
Status update: The patches for the last two remaining bugs are ready. One adds some missing keywords, another fixes bootstrapping with GCC 12.2 and unmasks the ebuild. The pull requests are https://github.com/gentoo/gentoo/pull/30730 and https://github.com/gentoo/prefix/pull/26.

Please take a review and post any comment or further discussion about both issues in their respective bug report, which are Bug 895334 and Bug 904474.
Comment 32 Tom Li 2023-04-24 21:23:44 UTC
Please make this bug DEPENDS ON Bug 898610 for cross-reference, it appears that we're still having problems on x64-macos as well.
Comment 33 Larry the Git Cow gentoo-dev 2023-04-25 15:10:09 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=c9f40c64294b16646560d39ae010fc18c6f28985

commit c9f40c64294b16646560d39ae010fc18c6f28985
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2023-04-25 15:06:30 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2023-04-25 15:06:30 +0000

    scripts/bootstrap-prefix: bump bootstrap snapshot for Darwin (arm64) fixes
    
    Closes: https://bugs.gentoo.org/898610
    Bug: https://bugs.gentoo.org/886491
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 34 Fabian Groffen gentoo-dev 2023-05-02 06:37:41 UTC
I have the impression a python-3.11 bump got in the way of completing a bootstrap post-tree-sync.
Comment 35 Tom Li 2023-05-02 19:09:39 UTC
I'm still waiting for Pull Request #2 "[cctools] fix ar(1) crash without argument" to be merged into grobian/darwin-xtools, so that a new release can be made for binutils-apple to fix Bug 905131.
Comment 36 Tom Li 2023-05-02 20:52:40 UTC
(In reply to Fabian Groffen from comment #34)
> I have the impression a python-3.11 bump got in the way of completing a
> bootstrap post-tree-sync.

The Python 3.11 has more problems than a tree import or adding keywords. It now depends on util-linux, making it unusable on macOS. I just opened a new Bug 905618 to discuss this problem.
Comment 37 Tom Li 2023-05-02 21:10:14 UTC
For missing Python keywords, I also opened a separate issue at Bug 905619.
Comment 38 Larry the Git Cow gentoo-dev 2023-05-05 07:05:29 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=40796bc559cfb718a27741290ddb878a125971b4

commit 40796bc559cfb718a27741290ddb878a125971b4
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2023-05-05 07:02:39 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2023-05-05 07:02:39 +0000

    dev-lang/python-3.11.3: bump patchset for upstream macOS GCC fix
    
    Closes: https://bugs.gentoo.org/886491
    Closes: https://github.com/gentoo/prefix/pull/29
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 dev-lang/python/Manifest             | 2 +-
 dev-lang/python/python-3.11.3.ebuild | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 39 Fabian Groffen gentoo-dev 2023-05-10 06:50:39 UTC
At least x86_64 bootstrap succeeded on Ventura:
https://bootstrap.prefix.bitzolder.nl/results/x86_64-apple-darwin22/20230509/
Comment 40 Fabian Groffen gentoo-dev 2023-05-10 10:02:47 UTC
fails on arm64 in stage3 on downloading distfiles.  Seems like openssl is foobared or something.

>>> Downloading 'https://downloads.sourceforge.net/lzmautils/xz-5.4.3.tar.gz'
--2023-05-10 11:51:54--  https://downloads.sourceforge.net/lzmautils/xz-5.4.3.  tar.gz
Auto configuration failed
8492424000:error:25FFF06C:DSO support routines:CRYPTO_internal:functionality    not supported:dso/dso_lib.c:226:
8492424000:error:0EFFF06E:configuration file routines:CRYPTO_internal:error     loading dso:conf/conf_mod.c:273:module=providers, path=providers
8492424000:error:0EFFF071:configuration file routines:CRYPTO_internal:unknown   module name:conf/conf_mod.c:214:module=providers
Comment 41 Tom Li 2023-06-04 11:27:01 UTC
(In reply to Fabian Groffen from comment #40)
> fails on arm64 in stage3 on downloading distfiles.  Seems like openssl is
> foobared or something.
> 
> >>> Downloading 'https://downloads.sourceforge.net/lzmautils/xz-5.4.3.tar.gz'
> --2023-05-10 11:51:54-- 
> https://downloads.sourceforge.net/lzmautils/xz-5.4.3.  tar.gz
> Auto configuration failed
> 8492424000:error:25FFF06C:DSO support routines:CRYPTO_internal:functionality
> not supported:dso/dso_lib.c:226:
> 8492424000:error:0EFFF06E:configuration file routines:CRYPTO_internal:error 
> loading dso:conf/conf_mod.c:273:module=providers, path=providers
> 8492424000:error:0EFFF071:configuration file
> routines:CRYPTO_internal:unknown   module
> name:conf/conf_mod.c:214:module=providers

Retested today on both ARM64 and x64, bootstrap on both machines were successful and I did not see any problem.

But I have seen this error message before elsewhere, I believe it was environmental-variable related. In a finished Gentoo prefix installation, if I use anything (e.g. "wget") based on OpenSSL in the Prefix without using the "startprefix" script, it's completely broken, with the error message "unknown module providers". The error disappears after "startprefix". So I conclude it was some kind of conflict/incompatibility between the macOS OpenSSL and Gentoo Prefix OpenSSL.

I cannot reproduce the problem anymore, however...
Comment 42 Fabian Groffen gentoo-dev 2023-06-04 12:02:45 UTC
Ok, shall we close this bug, as we both verified on arm64 and x64 that bootstrapping now works?
Comment 43 Tom Li 2023-06-06 19:35:13 UTC
(In reply to Fabian Groffen from comment #42)
> Ok, shall we close this bug, as we both verified on arm64 and x64 that
> bootstrapping now works?

+1. Let's close it.

If more problems arise in the future, it can be reopened later.
Comment 44 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-06-06 20:17:49 UTC
Many thanks for the hard work.