When trying to bootstrap Gentoo Prefix on macOS 10.15.5, it gives up quickly because /usr/include is missing: $ ./bootstrap-prefix.sh ... I'm excited! Seems we can finally do something productive now. Ok, I'm going to do a little bit of guesswork here. Thing is, your machine appears to be identified by CHOST=x86_64-apple-darwin19. Great! You appear to have a compiler in your PATH You don't have /usr/include, this thwarts me to build stuff. Please execute: xcode-select --install or install /usr/include in another way and try running me again. The suggestion doesn't help at all: $ xcode-select --install xcode-select: error: command line tools are already installed, use "Software Update" to install updates Reproducible: Always Steps to Reproduce: 1. curl -OL https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts/bootstrap-prefix.sh 2. ./bootstrap-prefix.sh Actual Results: bootstrap fails because /usr/include is missing Expected Results: bootstrap continues
Thanks, this is an ongoing problem for a while now. Unfortunately there is no workaround or solution at this point.
Is there more information on what the problem is exactly? Why is /usr/include required? I'm willing to take a look into this myself if I can find the time for it (but probably not before August, at best).
The problem is in a nutshell that the linker (binutils-apple) that we build is unable to deal with the objects created by the compiler provided by latest Xcode. On top of that, we cannot build llvm/clang in the right way yet where it considers the $EPREFIX/usr/include by default (in its searchpath). Using sysroot doesn't work, because that makes clang look inside that sysroot, and $EPREFIX is outside of it. I got GCC 9/10 working properly in this regard, so the idea is now to get a linker (might use the native linker for the time being) and GCC-10 to compile most stuff. The problem however is, when blocks support is necessary (as in the system frameworks). So the quest doesn't end there. Either we need block support for GCC, or find a way to patch llvm/clang to work in a mode like GCC-10 now works (basically oversimplified: see system headers from a given location, and append EPREFIX headers to that). The current approach using GCC I've made is to have a symlink in $EPREFIX called MacOSX.sdk pointing to the sdk in use (xcrun --show-sdk-path --sdk macosx). Then have GCC embed this path, such that we can switch the sdk without having to change the compiler. We tell GCC to look inside it using its --with-native-system-header-dir="${EPREFIX}/MacOSX.sdk/usr/include" configure argument. The existing approach using --with-local-prefix="${EPREFIX}/usr" makes sure we also include the EPREFIX installed headers. Since we don't have TAPI working (text files describing the API of libs that Apple uses in the Frameworks) our linker doesn't understand these tbd files, but looks directly in /usr/lib where the libraries all reside (the sysroot doesn't contain them, only the text references to the libs in /usr/lib). This actually works fine. Prefix has no real aim to be release independent or something.
This might be just wishful thinking on my part, but could we go the stage3 tarball route here? Basically, build a Prefix stage3 tarball on a High Sierra VM, test it on Catalina and Big Sur, then make it available? I know it just working around the problem, but I don't think leveraging the Xcode toolchain is going to get any easier going forward.
(In reply to Jason Cooper from comment #4) > then make it available? This seems to be explicitly forbidden by Apple SLA, item 4.B, at least in the current form where patches are applied to Apple software.
Which ebuilds in portage are "Apple software"? Can we replace them, or do something like: ``` cd gentoo tar -xjvf gentoo-prefix-stage3.tar.bz2 ./startprefix emerge --sync emerge --oneshot apple_productA apple_productB ``` ?
(In reply to Jason Cooper from comment #6) > Which ebuilds in portage are "Apple software"? Binutils-apple, tapi are the examples. Binutils-apple is a merge of 4 upstream packages -- ld64, cctools, dyld, libunwind. > Can we replace them, or do something like: I don't know but I guess that if we could, the big problem wouldn't have existed in the first place. I only begun to plunge into Apple's ecosystem so I may be ignorant about many details. But what I see is that bootstrapping is going relatively painless to the point of rebuilding toolchain (including libcxx and friends) and binutils in stage3 where numerous linking errors occur (caused by configure step referencing non-existent libraries) which I don't yet understand. And what is worse the errors seem to be not quite deterministic (sometimes they are different in several consecutive runs). Without these packages (using the Xcode's provided stuff) the rest seem to build relatively smooth.
(In reply to Alexander Bezrukov from comment #7) > (In reply to Jason Cooper from comment #6) > > Which ebuilds in portage are "Apple software"? > > Binutils-apple, tapi are the examples. Binutils-apple is a merge of 4 > upstream packages -- ld64, cctools, dyld, libunwind. > > > Can we replace them, or do something like: > > I don't know but I guess that if we could, the big problem wouldn't have > existed in the first place. I only begun to plunge into Apple's ecosystem so > I may be ignorant about many details. But what I see is that bootstrapping > is going relatively painless to the point of rebuilding toolchain (including > libcxx and friends) and binutils in stage3 where numerous linking errors > occur (caused by configure step referencing non-existent libraries) which I > don't yet understand. And what is worse the errors seem to be not quite > deterministic (sometimes they are different in several consecutive runs). > Without these packages (using the Xcode's provided stuff) the rest seem to > build relatively smooth. I think this is where https://bugs.gentoo.org/751412#c3 comes in (and I think bug 693444 is kind of getting at that).
(In reply to Sam James from comment #8) > think bug 693444 is kind of getting at that). Bug 693444 looks like a kind of memory corruption (or rather depending on uninitialized memory) in clang. I reduced the testcase to a single "#include <stdio.h>" and then did some experimentation and found that the success of compilation depends on the length of the command line and the current working directory. I was able to pass through this failure just (for example) by changing to .. and then removing initial .. from the path in -I option. But because of other errors farther on the way ("malformed object file" errors because just-compiled ld64 doesn't understand modern archives which come with recent Xcode) I just used the ld64 and other binaries from Xcode command line tools installation (I know, this is not Gentoo way). I also tried to upgrade binutils-apple to a recent version and even rebased patches to ld64-409.12 (I don't remember why I picked this version) but found that the effort needed is higher than I can afford at the moment.
(In reply to Alexander Bezrukov from comment #9) > (In reply to Sam James from comment #8) > > > think bug 693444 is kind of getting at that). > > Bug 693444 looks like a kind of memory corruption (or rather depending on > uninitialized memory) in clang. I reduced the testcase to a single "#include > <stdio.h>" and then did some experimentation and found that the success of > compilation depends on the length of the command line and the current > working directory. I was able to pass through this failure just (for > example) by changing to .. and then removing initial .. from the path in -I > option. But because of other errors farther on the way ("malformed object > file" errors because just-compiled ld64 doesn't understand modern archives > which come with recent Xcode) I just used the ld64 and other binaries from > Xcode command line tools installation (I know, this is not Gentoo way). I > also tried to upgrade binutils-apple to a recent version and even rebased > patches to ld64-409.12 (I don't remember why I picked this version) but > found that the effort needed is higher than I can afford at the moment. Right, so native-cctools package might be a solution here. I'm considering if we can use that for the time being to at least compile stuff.
(In reply to Fabian Groffen from comment #3) > I got GCC 9/10 working properly in this regard, so the idea is now to get a > linker (might use the native linker for the time being) and GCC-10 to > compile most stuff. The problem however is, when blocks support is > necessary (as in the system frameworks). So the quest doesn't end there. > Either we need block support for GCC, or find a way to patch llvm/clang to > work in a mode like GCC-10 now works (basically oversimplified: see system > headers from a given location, and append EPREFIX headers to that). > > The current approach using GCC I've made is to have a symlink in $EPREFIX > called MacOSX.sdk pointing to the sdk in use (xcrun --show-sdk-path --sdk > macosx). Then have GCC embed this path, such that we can switch the sdk > without having to change the compiler. We tell GCC to look inside it using > its --with-native-system-header-dir="${EPREFIX}/MacOSX.sdk/usr/include" > configure argument. The existing approach using > --with-local-prefix="${EPREFIX}/usr" makes sure we also include the EPREFIX > installed headers. Do you have a copy of bootstrap-prefix.sh that includes these changes? Could you please share? I would love to see how others have already solved many of the issues I'm running across so I can get up to speed and (hopefully) help with the problems you are dealing with.
(In reply to Fabian Groffen from comment #3) > On top of that, we cannot build llvm/clang in the right way > yet where it considers the $EPREFIX/usr/include by default (in its > searchpath). Using sysroot doesn't work, because that makes clang look > inside that sysroot, and $EPREFIX is outside of it. Could we patch clang? It looks like all of the path lookups for Darwin are in the Darwin driver. For c++ headers, it assumes that it should look alongside the clang binary or in <sysroot>. We could add a section before or after <sysroot> (or replace the <sysroot> section) that checks under $EPREFIX/MacOSX.sdk Here's the part of code in master (using a commit hash to allow safe linking to the lines) and llvmorg-12-init branches: master branch: https://github.com/llvm/llvm-project/blob/a3a24316087d0e1b4db0b8fee19cdee8b7968032/clang/lib/Driver/ToolChains/Darwin.cpp#L2033-L2049 llvmorg-12-init branch: https://github.com/llvm/llvm-project/blob/8c4a65b9b2ca6961139beca92de37eea479f00fa/clang/lib/Driver/ToolChains/Darwin.cpp#L2024-L2038 Similarly, for c headers: master branch: https://github.com/llvm/llvm-project/blob/a3a24316087d0e1b4db0b8fee19cdee8b7968032/clang/lib/Driver/ToolChains/Darwin.cpp#L1939-L1971 llvm-12-init branch: https://github.com/llvm/llvm-project/blob/8c4a65b9b2ca6961139beca92de37eea479f00fa/clang/lib/Driver/ToolChains/Darwin.cpp#L1940-L1972 And this path might need to be adjusted: master branch: https://github.com/llvm/llvm-project/blob/a3a24316087d0e1b4db0b8fee19cdee8b7968032/clang/lib/Driver/ToolChains/Darwin.cpp#L2063-L2064 llvmorg-12-init branch: https://github.com/llvm/llvm-project/blob/8c4a65b9b2ca6961139beca92de37eea479f00fa/clang/lib/Driver/ToolChains/Darwin.cpp#L2043-L2044 Plus a few other paths that might need to be adjusted (look for llvm::sys::path::append). Fabian, have you already gone down this route? Is this a dead end?
For C headers it looks like we can also replace /usr/include search path with something like: ./configure --with-c-include-dirs $EPREFIX/usr/include:$EPREFIX/MacOSX.sdk/usr/include In the clang code, this sets C_INCLUDE_DIRS. From a quick search we should be able to set this when building any version of clang. So, maybe we would only need to patch in support for c++ search paths if those need to be adjusted.
Sorry for so many comments. This is my last one until I hear back. :) I'm trying to compile the things I find out so that others don't need to go over the same research. I tried to find out what blocks are and what the status is based on these comments (and some helpful hints from sam_ in #gentoo-prefix). So, looking up blocks, I found the docs/specs for it (it's in docs as far back as clang 4.0.0): https://clang.llvm.org/docs/BlockLanguageSpec.html https://clang.llvm.org/docs/Block-ABI-Apple.html (In reply to Fabian Groffen from comment #3) > I got GCC 9/10 working properly in this regard, so the idea is now to get a > linker (might use the native linker for the time being) and GCC-10 to > compile most stuff. The problem however is, when blocks support is > necessary (as in the system frameworks). So the quest doesn't end there. > Either we need block support for GCC, or find a way to patch llvm/clang to > work in a mode like GCC-10 now works (basically oversimplified: see system > headers from a given location, and append EPREFIX headers to that). Sam linked to https://bugs.gentoo.org/751412#c3 where Fabian said: > If you can find a way to configure clang/llvm to "see" $EPREFIX/usr as primary > source of include and lib, then there is a way to move forward with clang. > > Because I couldn't find this, I tried to go with GCC, which works fine, and I > even had a very nice and clean path, but I'm still stuck with getting blocks > support working. > > I'll probably commit my pending blocks work someday soon, and try and see if I > can get help from GCC community, for I don't really get the errors/problems I > have with it, and need help should I want to finish this path. Which brings me to the gcc bug report requesting blocks support: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78352 Apple's gcc 4.2.1 block support changes are apparently not acceptable license-wise and technically too far removed from the current code base to be useful. And the last word there is: > Sorry that there hasn't been much progress on this - > it *was* top of my GCC11 TODO list, and then Apple Si. > came along and torpedoed that... So, getting GCC support for blocks is an excellent long term strategy, but it's not going to get Catalina support any time soon. So, it sounds like configuring or modifying clang is the way to go.
Uhm. I know I said that was my last comment. Sorry. more info. I glanced through the macports patches for llvm/clang to see if anything looked similar to what we're doing. These 3 looked interesting: https://github.com/macports/macports-ports/blob/master/lang/llvm-10/files/9001-macports-libstdcxx.diff https://github.com/macports/macports-ports/blob/master/lang/llvm-10/files/clang-driver-support-11.x-target.diff https://github.com/macports/macports-ports/blob/master/lang/llvm-10/files/5000-patch-compilerrtdarwinutils-find-macosxsdkversion.diff That last one shows that there are some cmake functions to find the MacOSX.sdk, so that may be somewhere else we need to modify: https://github.com/llvm/llvm-project/blob/master/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake
For prefix-standalone on Linux we had to patch Clang to make it work properly. Maybe some of those patches may also be useful to make Clang work properly on Macs. Have a look at lines 10-70 here: https://gitweb.gentoo.org/repo/gentoo.git/tree/profiles/features/prefix/standalone/profile.bashrc It would be great to get bootstrapping working again on Mac OS. That's what got me started as a dev in Gentoo :-) I was using prefix on my iMac. These days I use only Linux, and use prefix on Scientific Linux 6 and CentOS 7.
For sci-physics/root to work on prefix-standalone (it has llvm/clang bundled to make a C++ interpreter), I add the option -DDEFAULT_SYSROOT=${EPREFIX} to pick up headers from the prefix. Something like that will probably be necessary for Clang on Mac OS as well.
(In reply to Jacob Floyd from comment #11) > Do you have a copy of bootstrap-prefix.sh that includes these changes? Could > you please share? I would love to see how others have already solved many of > the issues I'm running across so I can get up to speed and (hopefully) help > with the problems you are dealing with. I don't have issues getting through stage1, stage2 is problematic though (xattr problems, odd enough). I've committed my changes (and a bump of the bootstrap snapshot so gentoo-functions is fixed) in 8559e1c3e7 (repo/proj/prefix.git)
and you probably want commits fd2058cbd9c38988b7b58e78f1c1d214273f329c, 2185ae8dc748888b8f86221018b77d4aac8770e8 and 224ac9a7692fc0cc66fe4fc2e63eed6596663c9f too.
It was a bit disturbing, actually. I had *JUST* made a hack to the prefix Portage locally to bypass the xattr issue and Zac committed a fix within ~30s of me writing it. It turns out you'd reported (bug 755950) it already ;) I'm now able to get to stage 3 where GCC failed but I'm retrying with your latest changes (I have a feeling my bootstrap was somehow broken b/c some hacks I had), with the /usr/include changes in: https://gist.github.com/cognifloyd/f4ff46e3dcfd48ca9827c32e04a8fe90#file-bootstrap-prefix-sh-L250-L256 https://gist.github.com/cognifloyd/f4ff46e3dcfd48ca9827c32e04a8fe90#file-bootstrap-prefix-sh-L2537 https://gist.github.com/cognifloyd/f4ff46e3dcfd48ca9827c32e04a8fe90#file-bootstrap-prefix-sh-L95-L98
Oh, I did hit some issues with it looking for *19-ar, but I'm wondering if that'll happen again. It triggers the macOS prompt for downloading the xcode CLI tools but I already had them - just not the ones it needed.
(In reply to Sam James from comment #21) > Oh, I did hit some issues with it looking for *19-ar, but I'm wondering if > that'll happen again. It triggers the macOS prompt for downloading the xcode > CLI tools but I already had them - just not the ones it needed. Given we're using the 10.14 profile for >= Mojave, this seems to be problematic: >* Compiling grep >./configure --host=x86_64-apple-darwin20 --prefix=/Users/sam/gentoo/tmp/usr --mandir=/Users/sam/gentoo/tmp/usr/share/man --infodir=/Users/sam/gentoo/tmp/usr/share/info --datadir=/Users/sam/gentoo/tmp/usr/share --sysconfdir=/Users/sam/gentoo/tmp/etc --localstatedir=/Users/sam/gentoo/tmp/var/lib --build=x86_64-apple-darwin20 --disable-perl-regexp checking for a BSD-compatible install... /Users/sam/gentoo/tmp/usr/bin/install -c It's using --host=x86_64-apple-darwin20 (and --build) but that's not the CHOST we're using so libraries aren't necessarily going to be available later on. I think this might be why my GCC built went awry.
However, I've only got x86_64-apple-darwin19 installed on my system, even with the latest Xcode 12.3 CLI tools beta. But the profile [0] wants a CHOST of x86_64-apple-darwin18. [0] https://github.com/gentoo/gentoo/blob/master/profiles/prefix/darwin/macos/10.14/x64/make.defaults
I added a change to hack in the right CHOST into make.profile. https://gist.github.com/cognifloyd/f4ff46e3dcfd48ca9827c32e04a8fe90#file-bootstrap-prefix-sh-L561-L564 > if [[ ${CHOST} == x86_64-apple-darwin19 ]]; then > # temp hack to fix CHOST until the macos/15.0 profile is in the tree > sed -i 's/darwin18/darwin19/' "${ROOT}"/etc/portage/make.profile/make.defaults > fi
FYI: I think I worked my way through a bunch of hoops, working my way up. I started on 10.13, my bootstrap is currently running in emerge -e system phase, 67 out of 83, so high likelyhood to succeed. I will continue on 10.14, hopefully not dealing with too many issues more. Then going on to Big Sur 11. I'm aiming at doing a GCC bootstrap in all cases, such that we can from there try and see if we can get llvm/clang in shape. I've applied a bunch of masks/changes in the profiles to make this possible, added a few keywords, and need to apply a couple of fixes still (most notably python-3.8 xattr module). I will require a bootstrap snapshot update, hopefully tomorrow's one is good, but I suspect we'll have a need for a few more in a row before all is stable and out of the box. I'm currently using DARWIN_USE_GCC=1 with bootstrap-prefix.sh, but as I said, it requires a newer snapshot, most notably for profiles, so I'd say don't bother trying it. Thanks for all the traction.
correction, I don't have 10.14 anymore, so will move forwards with 10.15.
Thanks Fabian. I won't try for a day or two (past midnight) so I get the snapshot with any extra things you think of. I'm hoping that some of the fixes handle the GCC build error I had (bug 756160) - nothing stands out to me in the recent changes. Maybe the new profile (with a consistent CHOST and deployment target) will help.
For clang >=10, there is a way to specify where the c++ stdlib headers are without setting -isysroot or setting SDKROOT (which the darwin driver passes treats as -isysroot). https://releases.llvm.org/10.0.0/tools/clang/docs/ClangCommandLineReference.html#cmdoption-clang1-stdlib-isystem-directory So, we can do something like this in bootstrap-prefix.sh: > configure_cflags() { > export CPPFLAGS="-isystem${ROOT}/tmp/usr/include" > if [[ ${CHOST} == *-darwin19 ]]; then > export CPPFLAGS+=" -isystem${ROOT}/MacOSX.sdk/usr/include" > # we require command line tools. This is not in the SDK. > export CPPFLAGS+=" -stdlib++-isystem/Library/Developer/CommandLineTools/usr/include/c++/v1" > fi And something similar under configure_toolchain. Checking for darwin19 is not very clean as it is neither forward nor backwards compatible, but that's all I have to test on. :) This seems to be working well. clang 10+ is in Xcode 10+ which requires at least OSX 10.13.6+ (High Sierra).
(In reply to Jacob Floyd from comment #28) > For clang >=10, there is a way to specify where the c++ stdlib headers are > without setting -isysroot or setting SDKROOT (which the darwin driver passes > treats as -isysroot). > > https://releases.llvm.org/10.0.0/tools/clang/docs/ClangCommandLineReference. > html#cmdoption-clang1-stdlib-isystem-directory > > So, we can do something like this in bootstrap-prefix.sh: > > configure_cflags() { > > export CPPFLAGS="-isystem${ROOT}/tmp/usr/include" > > if [[ ${CHOST} == *-darwin19 ]]; then > > export CPPFLAGS+=" -isystem${ROOT}/MacOSX.sdk/usr/include" > > # we require command line tools. This is not in the SDK. > > export CPPFLAGS+=" -stdlib++-isystem/Library/Developer/CommandLineTools/usr/include/c++/v1" > > fi > > And something similar under configure_toolchain. > > Checking for darwin19 is not very clean as it is neither forward nor > backwards compatible, but that's all I have to test on. :) > > This seems to be working well. > > clang 10+ is in Xcode 10+ which requires at least OSX 10.13.6+ (High Sierra). The CMake build system has an option called DEFAULT_SYSROOT which I'm making use of in prefix on Linux. I pass -DDEFAULT_SYSROOT="${EPREFIX}" when configuring to prevent clang from trying to use /usr/include.
(In reply to Sam James from comment #27) > Thanks Fabian. I won't try for a day or two (past midnight) so I get the > snapshot with any extra things you think of. > > I'm hoping that some of the fixes handle the GCC build error I had (bug > 756160) - nothing stands out to me in the recent changes. Maybe the new > profile (with a consistent CHOST and deployment target) will help. It seems I got a botched gcc-10.1.0 in the snapshot, so will have to wait for tomorrow to get a snapshot with a fixed version. Manually fixed snapshot in progress on 10.13 and 10.15 now.
my updates: 10.13: I have a problem with Frameworks/Security that exposes with gnutls, which apparently is in @system, but if one skips that, bootstrapping using GCC works 10.15: ensure you get latest 10.15 MacOSX SDK, then bootstrapping succeeds well past gcc, running into final emerge -e system, relatively postive in other words.
10.15: stops at the same point for gnutls, need header fix (same as 10.13 now) 11.0: did some fixes to bootstrap script to finish stage1, now need tree updates, e.g. bash-5.1_rc3 as nothing before that compiles on Big Sur
(In reply to Fabian Groffen from comment #32) > 10.15: stops at the same point for gnutls, need header fix (same as 10.13 > now) > 11.0: did some fixes to bootstrap script to finish stage1, now need tree > updates, e.g. bash-5.1_rc3 as nothing before that compiles on Big Sur Yeah, I keyworded locally for that for now (filed a bug before which has the patch). Looks like it got added to the script.
10.13: in the progress of a bootstrap now, hopefully to succeed after fix to gnutls (using GCC) 10.15: will restart it once 10.13 successfully bootstraps 11.0: with latest binutils-config-5.1-r3, GCC compilation seems to get past the problematic point of linking with -lc, it's building C++ standard libraries now
(In reply to Fabian Groffen from comment #34) > 10.13: in the progress of a bootstrap now, hopefully to succeed after fix to > gnutls (using GCC) > 10.15: will restart it once 10.13 successfully bootstraps > 11.0: with latest binutils-config-5.1-r3, GCC compilation seems to get past > the problematic point of linking with -lc, it's building C++ standard > libraries now Just got to stage3. GCC built fine in stage2, it's now rebuilding it in stage3 (11.0).
10.15: much like 11.0 needs bash-5.1 etc. efforts are ongoing 11.0: need to sort our an issue in Portage about it not being able to check install_names
10.15: emerge -e system stage, so likely fine 11.0: emerge -e system stage using todays bootstrap-prefix.sh so likely fine at last
11.0: as per http://bootstrap.prefix.bitzolder.nl/results/ success
I've defaulted the macOS bootstraps to GCC builds for now to allow people to bootstrap again, as this path is working on 10.13, 10.15 and 11.0. Thanks all for the help and input.
(In reply to Fabian Groffen from comment #39) > I've defaulted the macOS bootstraps to GCC builds for now to allow people to > bootstrap again, as this path is working on 10.13, 10.15 and 11.0. > > Thanks all for the help and input. Thanks a bunch Fabian. To continued work together, I hope.
