Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 701720 - Gentoo Prefix can't bootstrap on OSX 10.15.1
Summary: Gentoo Prefix can't bootstrap on OSX 10.15.1
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: AMD64 OS X
: Normal major
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords: Bug
Depends on:
Blocks: 755644
  Show dependency tree
 
Reported: 2019-12-02 05:57 UTC by wgjak47
Modified: 2020-12-01 10:21 UTC (History)
9 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
sys-apps/diffutils-3.7 failure on macOS prefix (stage2 bootstrap) (file_701720.txt,143.34 KB, text/plain)
2020-03-22 21:49 UTC, Sam James
Details
sys-devel/binutils-apple failure (file_701720.txt,185.81 KB, text/plain)
2020-03-22 22:51 UTC, Sam James
Details
Tarball containing stage1.log and text from the primary error (macos-gentoo-prefix-bootstrap-failure-25102020-01.tar.bz2,79.84 KB, application/x-bzip2)
2020-10-25 18:02 UTC, Robin Randhawa
Details

Note You need to log in before you can comment on or make changes to this bug.
Description wgjak47 2019-12-02 05:57:31 UTC
In Mac OS 10.15.1 the /usr/include was removed. And can't be install because root is write only now. refer: https://support.apple.com/en-us/HT210650. It seems that apple has move this to `xcrun --show-sdk-path`/usr/include

Could change path for bootstrap script work for this? I'm not good with shell programming and failed to modify it.

Reproducible: Always

Steps to Reproduce:
1. curl https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts/bootstrap-prefix.sh | bash -
Actual Results:  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 96094  100 96094    0     0  21949      0  0:00:04  0:00:04 --:--:-- 21949


                                             .
       .vir.                                d$b
    .d$$$$$$b.    .cd$$b.     .d$$b.   d$$$$$$$$$$$b  .d$$b.      .d$$b.
    $$$$( )$$$b d$$$()$$$.   d$$$$$$$b Q$$$$$$$P$$$P.$$$$$$$b.  .$$$$$$$b.
    Q$$$$$$$$$$B$$$$$$$$P"  d$$$PQ$$$$b.   $$$$.   .$$$P' `$$$ .$$$P' `$$$
      "$$$$$$$P Q$$$$$$$b  d$$$P   Q$$$$b  $$$$b   $$$$b..d$$$ $$$$b..d$$$
     d$$$$$$P"   "$$$$$$$$ Q$$$     Q$$$$  $$$$$   `Q$$$$$$$P  `Q$$$$$$$P
    $$$$$$$P       `"""""   ""        ""   Q$$$P     "Q$$$P"     "Q$$$P"
    `Q$$P"                                  """

             Welcome to the Gentoo Prefix interactive installer!


    I will attempt to install Gentoo Prefix on your system.  To do so, I'll
    ask  you some questions first.    After that,  you'll have to  practise
    patience as your computer and I try to figure out a way to get a lot of
    software  packages  compiled.    If everything  goes according to plan,
    you'll end up with what we call  "a Prefix install",  but by that time,
    I'll tell you more.



It seems to me you are 'guanjie.wang' (1225730825), that looks cool to me.

I'm going to check for some variables in your environment now:
  it appears ASFLAGS is not set :)
  it appears CFLAGS is not set :)
  it appears CPPFLAGS is not set :)
  it appears CXXFLAGS is not set :)
  it appears DYLD_LIBRARY_PATH is not set :)
  it appears GREP_OPTIONS is not set :)
  it appears LDFLAGS is not set :)
  it appears LD_LIBRARY_PATH is not set :)
  it appears LIBPATH is not set :)
  it appears PERL_MM_OPT is not set :)
  it appears PERL5LIB is not set :)
  it appears PKG_CONFIG_PATH is not set :)
  it appears PYTHONPATH is not set :)
  it appears ROOT is not set :)
  it appears CPATH is not set :)
  it appears LIBRARY_PATH is not set :)

I'm excited!  Seems we can finally do something productive now.

Ok, I'm going to do a little bit of guesswork here.  Thing is, your
machine appears to be identified by CHOST=x86_64-apple-darwin19.

Great!  You appear to have a compiler in your PATH

You don't have /usr/include, this thwarts me to build stuff.
Please execute:
  xcode-select --install
or install /usr/include in another way and try running me again.

Expected Results:  
Bootstrap normally

MacBook Pro 15 2019

And Nix looks encounter same problem: https://github.com/NixOS/nix/issues/2925
Comment 1 irpatte 2019-12-18 17:00:53 UTC
You can mount root with "sudo mount -uw /" and create /usr/include in which case it will continue, however darwin19 is not yet supported by the bootstrap script.  You can edit boostrap-prefix.sh to make it think you're on 10.14.x but there are quite a few problems once it gets started running on 10.15.x such as libraries and headers not being in the right place (you have to symlink and create quite a lot of files).  

If you want to try it (it won't really work but it'll get you past where you are now) there's this in the script:

x86_64-apple-darwin9|x86_64-apple-darwin1[012345678])
			rev=${CHOST##*darwin}
			profile="prefix/darwin/macos/10.$((rev - 4))/x64"
			;;

under that add:

x86_64-apple-darwin19)
			profile="prefix/darwin/macos/10.14/x64
			;;

or at least that what worked for me.  But basically 10.15.x isn't supported yet so it's rather broken
Comment 2 Fabian Groffen gentoo-dev 2019-12-23 09:31:41 UTC
The road forward here is to get xnu-headers and darwin-libc-headers packages put somewhere such that system headers will be available (regardless of Xcode) and probably a bootstrap compiler to be installed as well.
Comment 3 Larry the Git Cow gentoo-dev 2019-12-25 15:33:50 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=91734750c199af8e90db3d62e20e45b94d20357c

commit 91734750c199af8e90db3d62e20e45b94d20357c
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2019-12-25 15:29:08 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2019-12-25 15:33:07 +0000

    scripts/bootstrap-prefix: use 10.14 profile for 10.15 and up
    
    We don't have sources available for 10.15 (yet), so let's build a
    10.14-based toolchain instead
    
    Bug: https://bugs.gentoo.org/701720
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 5 +++++
 1 file changed, 5 insertions(+)
Comment 4 Brad Laue 2020-03-02 03:48:20 UTC
XCode seems to have /usr/include in two places:

/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include

I note MacPorts using the former location with the -isysroot argument to GCC, so that /usr/include is derived from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk per https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html.

Anything we can do here? I'll play around and do some testing myself.
Comment 5 Fabian Groffen gentoo-dev 2020-03-02 07:44:11 UTC
the problem is mostly clang/llvm not playing well with the modifications we made (which made it work in the previous setting)

So once you get through the initial bootstrap_stage2, you end up with a compiler that won't cooperate.
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-03-22 21:48:49 UTC
Workarounds I've had to use on macOS 10.15.3 before each stage:
* pre-stage1: sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/ (or see link [0] below)
* pre-stage2: mkdir -p ${EPREFIX}/tmp/var/db/repos/ -p; ln -s ${EPREFIX}/var/db/repos/gentoo/ ${EPREFIX}/tmp/var/db/repos/gentoo
* pre-stage2: change 'Apple LLVM' to 'Apple clang' in the script for compiler detection

I also used bash instead of zsh for safekeeping, and as per the manual bootstrapping guide [1], set my PATH at the beginning:
> export PATH="${EPREFIX}/usr/bin:${EPREFIX}/bin:${EPREFIX}/tmp/usr/bin:${EPREFIX}/tmp/bin:/usr/bin:/bin"
otherwise the scripts can't find emerge going into stage2.

Using all of this together, I have got partway through stage2 compilation but it dies on coreutils right now (log attached).

[0] https://stackoverflow.com/a/58349403
[1] https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap
Comment 7 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-03-22 21:49:45 UTC
Created attachment 624610 [details]
sys-apps/diffutils-3.7 failure on macOS prefix (stage2 bootstrap)
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-03-22 22:50:39 UTC
(In reply to sam_c (Security Padawan) from comment #7)
> Created attachment 624610 [details]
> sys-apps/diffutils-3.7 failure on macOS prefix (stage2 bootstrap)

By patching out gnulib-tests from Makefile.in, I am able to get further. binutils-apple fails due to header errors like this: https://github.com/PointCloudLibrary/pcl/issues/2601#issuecomment-483517233.

I imagine this is the stuff grobian was warning about. Attached failure.
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-03-22 22:51:04 UTC
Created attachment 624656 [details]
sys-devel/binutils-apple failure
Comment 10 Jacob Floyd 2020-10-07 18:02:45 UTC
I just ran into this on MacOS Catalina 10.15.7

I was not able to do `sudo mount -uw /` so, instead I ran `sudo mkdir /usr/local/include` and modified line 2427 of bootstrap-prefix.sh:

```
         if type -P xcode-select > /dev/null ; then
-                if [[ ! -d /usr/include ]] ; then
+                if [[ ! -d /usr/local/include ]] ; then
                        # bug #512032
```

I'm now waiting for stage1 to complete.
Comment 11 Robin Randhawa 2020-10-25 17:33:01 UTC
@Jacob Floyd: Would be useful to know whether you were able to make further progress ? Thanks.
Comment 12 Robin Randhawa 2020-10-25 17:36:54 UTC
Given that this ticket is more than 9 months old, would it be fair to say that the Platform Matrix on https://wiki.gentoo.org/wiki/Project:Prefix is a bit misleading ?

Not meaning to antagonise: I really want to see this platform work out of the box! Happy to help as best as I can.
Comment 13 Robin Randhawa 2020-10-25 18:01:20 UTC
FWIW: On a whim, I followed Jacob's path of modding bootstrap-prefix.sh such that the script looks for /usr/local/include instead of /usr/include. To be honest, I suspected that this wouldn't work - surely there are essential headers expected within ? Creating a new dir like so without populating with the headers probably wouldn't work.

All the same: Things eventually bomb out while building gawk.

The key failure messages are:

=============
make[1]: Entering directory '/Users/jagran01/Gentoo/var/tmp/gawk-3.1.8/gawk-3.1.8'
Making all in .
make[2]: Entering directory '/Users/jagran01/Gentoo/var/tmp/gawk-3.1.8/gawk-3.1.8'
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT array.o -MD -MP -MF .deps/array.Tpo -c -o array.o array.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT awkgram.o -MD -MP -MF .deps/awkgram.Tpo -c -o awkgram.o awkgram.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT builtin.o -MD -MP -MF .deps/builtin.Tpo -c -o builtin.o builtin.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT dfa.o -MD -MP -MF .deps/dfa.Tpo -c -o dfa.o dfa.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT ext.o -MD -MP -MF .deps/ext.Tpo -c -o ext.o ext.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT field.o -MD -MP -MF .deps/field.Tpo -c -o field.o field.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT floatcomp.o -MD -MP -MF .deps/floatcomp.Tpo -c -o floatcomp.o floatcomp.c
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT gawkmisc.o -MD -MP -MF .deps/gawkmisc.Tpo -c -o gawkmisc.o gawkmisc.c
mv -f .deps/floatcomp.Tpo .deps/floatcomp.Po
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT getopt.o -MD -MP -MF .deps/getopt.Tpo -c -o getopt.o getopt.c
mv -f .deps/gawkmisc.Tpo .deps/gawkmisc.Po
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT getopt1.o -MD -MP -MF .deps/getopt1.Tpo -c -o getopt1.o getopt1.c
mv -f .deps/ext.Tpo .deps/ext.Po
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT io.o -MD -MP -MF .deps/io.Tpo -c -o io.o io.c
mv -f .deps/getopt1.Tpo .deps/getopt1.Po
gcc -DDEFPATH='".:/Users/jagran01/Gentoo/tmp/usr/share/awk"' -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/Users/jagran01/Gentoo/tmp/usr/share/locale"' -I.     -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c
io.c:1886:3: error: implicit declaration of function 'ioctl' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ioctl(slave, TIOCSCTTY, 0);
                ^
1 error generated.
==========

I'll upload the complete stage1.log file and a more detailed view of the gawk failure in an attachment.

Happy to follow leads. Thanks.
Comment 14 Robin Randhawa 2020-10-25 18:02:01 UTC
Created attachment 668507 [details]
Tarball containing stage1.log and text from the primary error
Comment 15 Fabian Groffen gentoo-dev 2020-10-25 19:04:38 UTC
(In reply to Robin Randhawa from comment #12)
> Given that this ticket is more than 9 months old, would it be fair to say
> that the Platform Matrix on https://wiki.gentoo.org/wiki/Project:Prefix is a
> bit misleading ?
> 
> Not meaning to antagonise: I really want to see this platform work out of
> the box! Happy to help as best as I can.

Yes, macOS is currently unsupported, except for people who managed to bootstrap before or somehow by advanced hacks.
Comment 16 Jacob Floyd 2020-10-26 00:13:47 UTC
Sadly, no. I was not able to get much farther. I ran out of time, so I started using macports instead.

I wonder what the next step to get this installed might be?
Maybe go back to a manual bootstrap, using the bootstrap script and the old prefix bootstrap docs as a guide?

It'll be awhile before I can revisit this and try again. :(
Comment 17 Jacob Floyd 2020-11-19 16:48:59 UTC
Based on https://bugs.gentoo.org/730476#c3
I modified bootstrap-prefix.sh to use headers from MacOSX.sdk.
I finally got past stage1 by version bumping a few packages, using gcc instead of clang for two packages, and by symlinking a macports installed m4 (I couldn't get a functional m4 even though compilation succeeded during bootstrap).

My changes to get through phase 1 are here (up through revision 7 of the gist):
https://gist.github.com/cognifloyd/f4ff46e3dcfd48ca9827c32e04a8fe90/e9461a5241b7bec0fce6ac31836551c7097110b6

It sounds like a variety of people have got past phase1. Could we commit changes to bootstrap-prefix.sh so that bootstrap phase1 can pass for everyone? It would be nice to not have everyone separately find hacks to get past that, so we can work together to get things working in phase2 and beyond.

Now phase 2 is failing to emerge the first package: sys-apps/gentoo-functions, so I started modifying a few other things, but it will be a bit before I can spend more time on hacking on that.

So, how do we go about getting changes included in bootstrap-prefix.sh to help with phase1?
Comment 18 Fabian Groffen gentoo-dev 2020-11-29 16:25:13 UTC
this works but only when DARWIN_USE_GCC=1 is exported during bootstrap
Comment 19 Fabian Groffen gentoo-dev 2020-12-01 10:21:00 UTC
OP: please try again using the latest bootstrap-prefix.sh script, this should be fixed.

Jacob: please file additional bugs for Clang/LLVM/binutils-apple fixes etc.  Let's try to tackle that one now we've got some ground under our feet again.