Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 778014 - Prefix: Big Sur ARM (M1 MacBook Pro MYD82LL/A) build failure due to missing symbols
Summary: Prefix: Big Sur ARM (M1 MacBook Pro MYD82LL/A) build failure due to missing s...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: ARM64 OS X
: Highest blocker with 5 votes (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords:
: 832037 863503 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-03-24 19:49 UTC by Kenneth G. Strawn
Modified: 2022-08-29 03:51 UTC (History)
9 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (build.log,395.52 KB, text/plain)
2021-03-24 19:50 UTC, Kenneth G. Strawn
Details
build logs (gcc-build-logs.tar.bz2,477.24 KB, application/x-bzip2)
2022-07-31 17:09 UTC, Askar Bektassov
Details
ERROR: app-admin/perl-cleaner-2.30::gentoo_prefix failed (build.log,2.88 KB, text/plain)
2022-08-04 14:46 UTC, Akase Cho
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kenneth G. Strawn 2021-03-24 19:49:44 UTC
bootstrap_prefix.sh at least appears to do fine up until the attempt to compile GCC, at which point numerous "undefined symbol" errors arise

Reproducible: Always

Steps to Reproduce:
1. wget https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts/bootstrap-prefix.sh
2. chmod +x bootstrap-prefix.sh
3. ./bootstrap-prefix.sh
Actual Results:  
Undefined symbols for architecture arm64:
  "__Z5yyendv", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
  "__Z5yylexPPKc", referenced from:
      __ZL5tokenv in gengtype-parse.o
  "__Z7yybeginPKc", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
  "_lexer_line", referenced from:
      __ZL22create_optional_field_P4pairP4typePKcS4_i in gengtype.o
      __ZL20adjust_field_rtx_defP4typeP7options in gengtype.o
      __ZL21adjust_field_tree_expP4typeP7options in gengtype.o
      __Z17adjust_field_typeP4typeP7options in gengtype.o
      __ZL11parse_errorPKcz in gengtype-parse.o
      __ZL16struct_field_seqv in gengtype-parse.o
      __ZL4typePP7optionsb in gengtype-parse.o
      ...
  "_lexer_toplevel_done", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:3004: build/gengtype] Error 1
make[3]: *** Waiting for unfinished jobs....
/opt/gentoo/tmp/bin/bash /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/../move-if-change tmp-options.h options.h
echo timestamp > s-options-h
/opt/gentoo/tmp/bin/bash /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/../move-if-change tmp-c-target-hooks-def.h \
				     c-family/c-target-hooks-def.h
/opt/gentoo/tmp/bin/bash /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/../move-if-change tmp-target-hooks-def.h \
				     target-hooks-def.h
/opt/gentoo/tmp/bin/bash /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/../move-if-change tmp-d-target-hooks-def.h \
				     d/d-target-hooks-def.h
/opt/gentoo/tmp/bin/bash /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/../move-if-change tmp-common-target-hooks-def.h \
				     common/common-target-hooks-def.h
echo timestamp > s-c-target-hooks-def-h
echo timestamp > s-target-hooks-def-h
echo timestamp > s-d-target-hooks-def-h
echo timestamp > s-common-target-hooks-def-h
rm gcov.pod lto-dump.pod gfdl.pod gpl.pod cpp.pod gcc.pod gcov-dump.pod gcov-tool.pod fsf-funding.pod
make[3]: Leaving directory '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/build/gcc'
make[2]: *** [Makefile:4786: all-stage1-gcc] Error 2
make[2]: Leaving directory '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/build'
make[1]: *** [Makefile:22346: stage1-bubble] Error 2
make[1]: Leaving directory '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/build'
make: *** [Makefile:1009: all] Error 2
 * ERROR: sys-devel/gcc-11_pre20200206::gentoo_prefix failed (compile phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=sys-devel/gcc-11_pre20200206::gentoo_prefix'`,
 * the complete build log and the output of `emerge -pqv '=sys-devel/gcc-11_pre20200206::gentoo_prefix'`.
 * The complete build log is located at '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/temp/build.log'.
 * The ebuild environment file is located at '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/temp/environment'.
 * Working directory: '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/build'
 * S: '/opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si'
 * 
 * Please include /opt/gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-build-logs.tar.bz2 in your bug report.
 * 

Expected Results:  
Successful build

Attaching full build.log file after publishing this
Comment 1 Kenneth G. Strawn 2021-03-24 19:50:30 UTC
Created attachment 693357 [details]
build.log

Log file for failed GCC build
Comment 2 Fabian Groffen gentoo-dev 2021-03-25 07:16:05 UTC
Yes, the problem is m4 calling flex.  For some reason execv() call fails, leaving an empty file, which is then ignored.  Obviously, from there you get these missing symbols.

There is nothing we/I can find as to why the exec call fails, nor why the version provided by Apple /does/ work.  Compiling their code results in the same non-functional binary.
Comment 3 Kenneth G. Strawn 2021-03-25 17:47:28 UTC
I wonder therefore if there's a way to patch M4 to make it use Apple's exec() call instead of M4's on arm64-macos-big-sur as a temporary workaround.
Comment 4 Kenneth G. Strawn 2021-03-25 17:48:32 UTC
Oops, bad wording: I wonder therefore if there's a way to patch M4 to use Apple's exec() call instead of Flex's on arm64-macos11 as a temporary workaround.
Comment 5 Fabian Groffen gentoo-dev 2021-03-25 18:18:09 UTC
perhaps that's an option for the time being, to see if there's other stuff popping up.

Sidenote, you can just bootstrap x86_64 mode if you need a working system: /usr/bin/arch -x86_64 ./bootstrap-prefix.
Comment 6 Steve Gilberd 2021-05-27 14:47:03 UTC
> the problem is m4 calling flex...

Shouldn't this read "flex calling m4"? That appears to be the issue on my system, and the symptoms are otherwise identical to this bug report.

Interestingly, while the exec call fails when flex is run via make, running exactly the same flex command from a shell instead completes without any errors, and results in a gengtype-lex.c which works correctly. Injecting this file into the build process, replacing the make-generated one, allows the gcc build to proceed properly, without encountering the missing symbol error.

```
TARGET_CPU_DEFAULT="" \                                            
HEADERS="auto-host.h ansidecl.h" DEFINES="" \
/Volumes/Gentoo/tmp/bin/bash /Volumes/Gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/mkconfig.sh config.h
TARGET_CPU_DEFAULT="" \
HEADERS="options.h insn-constants.h config/aarch64/biarchlp64.h config/aarch64/aarch64.h config/darwin.h config/aarch64/darwin.h config/aarch64/aarch64-errata.h config/initfini-array.h defaults.h" DEFINES="LIBC_GLIBC=1 LIBC_UCLIBC=2 LIBC_BIONIC=3 LIBC_MUSL=4 DEF_MIN_OSX_VERSION=\"11.0\" DEF_LD64=\"609.0\" TARGET_DEFAULT_ASYNC_UNWIND_TABLES=1" \
/Volumes/Gentoo/tmp/bin/bash /Volumes/Gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/mkconfig.sh tm.h
TARGET_CPU_DEFAULT="" \
HEADERS="config/aarch64/aarch64-protos.h config/arm/aarch-common-protos.h config/darwin-protos.h tm-preds.h" DEFINES="" \
/Volumes/Gentoo/tmp/bin/bash /Volumes/Gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/mkconfig.sh tm_p.h
TARGET_CPU_DEFAULT="" \
HEADERS="auto-host.h ansidecl.h" DEFINES="" \
/Volumes/Gentoo/tmp/bin/bash /Volumes/Gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/mkconfig.sh bconfig.h
flex  -ogengtype-lex.c /Volumes/Gentoo/var/tmp/portage/sys-devel/gcc-11_pre20200206/work/gcc-darwin-arm64-master-wip-apple-si/gcc/gengtype-lex.l && { \
  echo '#ifdef HOST_GENERATOR_FILE' > gengtype-lex.c.tmp; \
  echo '#include "config.h"'       >> gengtype-lex.c.tmp; \
  echo '#else'                     >> gengtype-lex.c.tmp; \
  echo '#include "bconfig.h"'      >> gengtype-lex.c.tmp; \
  echo '#endif'                    >> gengtype-lex.c.tmp; \
  cat gengtype-lex.c >> gengtype-lex.c.tmp; \
  mv gengtype-lex.c.tmp gengtype-lex.c; \
}
```
Comment 7 Fabian Groffen gentoo-dev 2021-05-27 18:01:52 UTC
yes, and the mystery remains why the call fails
Comment 8 Steve Gilberd 2021-06-03 08:35:38 UTC
The precise location this error ultimately seems to originate from is here:
https://github.com/westes/flex/blob/04c5b7c9209801aa1bdbf279ccdcde0d57874a55/src/filter.c#L182

Of particular note is the comment beginning on line 156 - it looks like there are hacky assumptions being made, which may (given that Mac OS is a BSD derivative) be applicable here, and not working as the author of that particular bit of code intended.
Comment 9 Fabian Groffen gentoo-dev 2021-06-03 08:41:25 UTC
Indeed, but even when you compile Apple's version of flex (from their opensource page), which actually does the same, it fails as well.

So, at least in my attempts, compiling Apple's version of flex using Apple's Clang, doesn't result in a working flex, whereas Apple's provided binary for flex works.

So it appears there's something we do not have here.  Just to be complete, the same OS version on x64, the whole problem isn't present, so it appears to be something entirely related to the CPU architecture.  Even using Rosetta this problem doesn't appear.
Comment 10 Steve Gilberd 2021-06-03 08:47:12 UTC
Interesting.

A bit of wild speculation here - do we know whether Apple's flex binary was cross-compiled, or compiled natively?

Do we know which version of Clang Apple used to build it?
Comment 11 Fabian Groffen gentoo-dev 2021-06-03 08:51:33 UTC
I don't think we can see any of that from the binary.  We know which version of Clang there was at the time of the introduction of the OS, but there's no guarantee that that's the version they used to build the OS.
Comment 12 Sylwester Arabas 2021-10-15 17:52:34 UTC
FWIW, just confirming that the problem has not disappeared, same symptoms here. Any hints? Thanks
Comment 13 john 2022-01-26 09:52:01 UTC
*** Bug 832037 has been marked as a duplicate of this bug. ***
Comment 14 john 2022-01-26 10:02:50 UTC
Also experiencing the same issue and erroneously filed 832037 before I stumbled on this one :)
Comment 15 Askar Bektassov 2022-07-31 17:09:22 UTC
Created attachment 796207 [details]
build logs
Comment 16 Askar Bektassov 2022-07-31 17:11:41 UTC
I assume gcc compilation is not fixed yet. Still failing stage2. Fabien, I noticed your sidenote, concerning bootstrapping in x86_64 mode (/usr/bin/arch -x86_64 ./bootstrap-prefix).

Assuming it works, what will be my steps once the GCC (and bootstrap process in general) has been fixed?
Comment 17 Fabian Groffen gentoo-dev 2022-08-01 11:34:18 UTC
you can work around this by removing flex from the prefix, then resuming the bootstrap
you'll need to repeat this when it re-emerges gcc

need to find a workaround so I make gcc not use flex from Prefix, perhaps as simple as setting FLEX or something.

Then you'll need to use -O1 at most, or force it for certain packages like groff and vim.

That should mostly get you up to speed with a native M1 Prefix.  It works for me pretty well, but haven't been able to remove the rough edges yet from the bootstrap process.
Comment 18 Askar Bektassov 2022-08-02 07:44:06 UTC
Hi Fabien, thanks for the usual assistance. I tried removing flex from prefix, without success.

Apparently, the issue is somewhat related to compiler's inability to locate gmp.h.

In file included from /Users/askarbektassov/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/gcc-12-branch-gcc-12.1-darwin-r0/gcc/config/aarch64/aarch64-speculation.cc:22:
/Users/askarbektassov/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/gcc-12-branch-gcc-12.1-darwin-r0/gcc/system.h:703:10: fatal error: 'gmp.h' file not found
#include <gmp.h>
         ^~~~~~~
1 error generated.
make[2]: *** [/Users/askarbektassov/Gentoo/tmp/var/tmp/portage/sys-devel/gcc-12.1.0/work/gcc-12-branch-gcc-12.1-darwin-r0/gcc/config/aarch64/t-aarch64:151: aarch64-speculation.o] Error 1
Comment 19 Fabian Groffen gentoo-dev 2022-08-02 08:28:10 UTC
Hmmm, that's odd.  Do you have $EPREFIX/tmp/usr/include/gmp.h?
Comment 20 Askar Bektassov 2022-08-02 09:46:38 UTC
Yep, it is there.

askarbektassov@MacBook-Pro ~ % ls $EPREFIX/tmp/usr/include/gmp.h  
/Users/askarbektassov/Gentoo/tmp/usr/include/gmp.h

I re downloaded bootstrap-prefix.sh, deleted Gentoo folder and run the bootstrap from scratch. The issue reproduces anyway.

It seems this is the issue preventing compilation of gcc on my machine. Cheers.
Comment 21 Askar Bektassov 2022-08-02 09:47:15 UTC
Yep, it is there.

askarbektassov@MacBook-Pro ~ % ls $EPREFIX/tmp/usr/include/gmp.h  
/Users/askarbektassov/Gentoo/tmp/usr/include/gmp.h

I re downloaded bootstrap-prefix.sh, deleted Gentoo folder and run the bootstrap from scratch. The issue reproduces anyway.

It seems this is the issue preventing compilation of gcc on my machine. Cheers.
Comment 22 Fabian Groffen gentoo-dev 2022-08-02 13:25:55 UTC
FWIW I pushed the workaround for flex to the gcc ebuild, but that one is not in a snapshot yet.  I'll try and see what it does tomorrow.
Comment 23 Larry the Git Cow gentoo-dev 2022-08-03 12:01:41 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=5025a353887de4e70e4ac960f9ec2051b387e1e2

commit 5025a353887de4e70e4ac960f9ec2051b387e1e2
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-03 11:58:41 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-03 12:01:35 +0000

    scripts/bootstrap-prefix: force CXXFLAGS for GCC to find our includes
    
    GCC apparently at some point stops looking at CPPFLAGS, which causes it
    not to "see" our includes.  So, use CXXFLAGS isntead while we don't have
    helpers to make this work.
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 4 ++++
 1 file changed, 4 insertions(+)
Comment 24 Larry the Git Cow gentoo-dev 2022-08-03 14:59:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e2526f187ac6aa04739b49de0f7c202ec9e7d92e

commit e2526f187ac6aa04739b49de0f7c202ec9e7d92e
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-03 14:56:54 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-03 14:59:21 +0000

    profiles/prefix/darwin/macos/*/arm64/gcc: use nold variant
    
    binutils-apple doesn't compile currently on arm64 (#792780) and needs
    work, use the host-provided linker for the time being.
    
    Closes: https://bugs.gentoo.org/792780
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 profiles/prefix/darwin/macos/11.0/arm64/gcc/parent | 2 +-
 profiles/prefix/darwin/macos/12.0/arm64/gcc/parent | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 25 Askar Bektassov 2022-08-04 06:21:10 UTC
Hey Fabian, your workaround allowed me to complete stage2, but then when progressing through stage3 the same issue described by OP comes back. Does this mean that after stage2 I should remove flex and compile gcc, as you suggested on august 1?


Undefined symbols for architecture arm64:
  "__Z5yyendv", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
  "__Z5yylexPPKc", referenced from:
      __ZL5tokenv in gengtype-parse.o
  "__Z7yybeginPKc", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
  "_lexer_line", referenced from:
      __ZL22create_optional_field_P4pairP4typePKcS4_i in gengtype.o
      __ZL20adjust_field_rtx_defP4typeP7options in gengtype.o
      __ZL21adjust_field_tree_expP4typeP7options in gengtype.o
      __Z17adjust_field_typeP4typeP7options in gengtype.o
      __ZL11parse_errorPKcz in gengtype-parse.o
      __ZL16struct_field_seqv in gengtype-parse.o
      __ZL4typePP7optionsb in gengtype-parse.o
      ...
  "_lexer_toplevel_done", referenced from:
      __Z10parse_filePKc in gengtype-parse.o
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status
Comment 26 Fabian Groffen gentoo-dev 2022-08-04 06:50:04 UTC
I finished a bootstrap yesterday, so with today's snapshot you should be able to progress (the GCC ebuild just forces the use of /usr/bin/flex to work around this issue)
Comment 27 Fabian Groffen gentoo-dev 2022-08-04 06:51:43 UTC
*** Bug 863503 has been marked as a duplicate of this bug. ***
Comment 28 Larry the Git Cow gentoo-dev 2022-08-04 11:51:02 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=662d541fbb2e2165e98e2a79556438ba7a95dd36

commit 662d541fbb2e2165e98e2a79556438ba7a95dd36
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-04 11:49:51 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-04 11:49:51 +0000

    scripts/bootstrap-prefix: bump snapshot for arm64-macos
    
    This should get past GCC in stage2, and depending on your machine,
    complete the entire bootstrap.
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
Comment 29 Akase Cho 2022-08-04 14:46:42 UTC
Created attachment 797668 [details]
ERROR: app-admin/perl-cleaner-2.30::gentoo_prefix failed

>>> Emerging (74 of 74) app-admin/perl-cleaner-2.30::gentoo_prefix
 * perl-cleaner-2.30.tar.bz2 BLAKE2B SHA512 size ;-) ...                 [ ok ]
>>> Unpacking source...
>>> Unpacking perl-cleaner-2.30.tar.bz2 to /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work
>>> Source unpacked in /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work
>>> Preparing source in /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work/perl-cleaner-2.30 ...
 * Adjusting to prefix /Users/lightsing/Gentoo
 *   perl-cleaner ...
 [ ok ]
>>> Source prepared.
sed: no input files
sed: no input files
>>> Configuring source in /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work/perl-cleaner-2.30 ...
>>> Source configured.
>>> Compiling source in /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work/perl-cleaner-2.30 ...
>>> Source compiled.
>>> Test phase [not enabled]: app-admin/perl-cleaner-2.30

>>> Install app-admin/perl-cleaner-2.30 into /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/image
>>> Completed installing app-admin/perl-cleaner-2.30 into /Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/image

 * Final size of build directory: 28 KiB
 * Final size of installed tree:  28 KiB

/Users/lightsing/Gentoo/usr/lib/python-exec/python3.9/portageq: this Python implementation (python3.9) is not supported by the script.
 * ERROR: app-admin/perl-cleaner-2.30::gentoo_prefix failed:
 *   has_version: unexpected portageq exit code: 127
 *
 * Call stack:
 *   misc-functions.sh, line 1334:  Called install_qa_check
 *   misc-functions.sh, line  138:  Called source 'install_symlink_html_docs'
 *        60python-pyc, line  118:  Called python_pyc_check
 *        60python-pyc, line   22:  Called has_version 'install_hooks'
 *    phase-helpers.sh, line  986:  Called ___best_version_and_has_version_common '/Users/lightsing/Gentoo/var/db/repos/gentoo/metadata/install-qa-check.d/60python-pyc'
 *    phase-helpers.sh, line  973:  Called die
 * The specific snippet of code:
 *   				die "${FUNCNAME[1]}: unexpected portageq exit code: ${retval}"
 *
 * If you need support, post the output of `emerge --info '=app-admin/perl-cleaner-2.30::gentoo_prefix'`,
 * the complete build log and the output of `emerge -pqv '=app-admin/perl-cleaner-2.30::gentoo_prefix'`.
 * The complete build log is located at '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/temp/build.log'.
 * The ebuild environment file is located at '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/temp/environment'.
 * Working directory: '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/image'
 * S: '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work/perl-cleaner-2.30'
!!! post install failed; exiting.

>>> Failed to emerge app-admin/perl-cleaner-2.30, Log file:

>>>  '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/temp/build.log'
 * Messages for package app-misc/ca-certificates-20211016.3.80:
 * You should run update-ca-certificates manually after etc-update
 * Messages for package sys-apps/portage-3.0.34.2:
 *
 * Detected file collision(s):
 *
 * 	/Users/lightsing/Gentoo/usr/share/portage/config/make.conf.example
 * 	/Users/lightsing/Gentoo/usr/share/portage/config/repos.conf
 * 	/Users/lightsing/Gentoo/usr/share/portage/config/make.globals
 * 	/Users/lightsing/Gentoo/usr/share/portage/config/sets/portage.conf
 *
 * Searching all installed packages for file collisions...
 *
 * Press Ctrl-C to Stop
 *
 * None of the installed packages claim the file(s).
 *
 * Package 'sys-apps/portage-3.0.34.2' merged despite file collisions. If
 * necessary, refer to your elog messages for the whole content of the
 * above message.
 * Messages for package app-admin/perl-cleaner-2.30:
 * ERROR: app-admin/perl-cleaner-2.30::gentoo_prefix failed:
 *   has_version: unexpected portageq exit code: 127
 *
 * Call stack:
 *   misc-functions.sh, line 1334:  Called install_qa_check
 *   misc-functions.sh, line  138:  Called source 'install_symlink_html_docs'
 *        60python-pyc, line  118:  Called python_pyc_check
 *        60python-pyc, line   22:  Called has_version 'install_hooks'
 *    phase-helpers.sh, line  986:  Called ___best_version_and_has_version_common '/Users/lightsing/Gentoo/var/db/repos/gentoo/metadata/install-qa-check.d/60python-pyc'
 *    phase-helpers.sh, line  973:  Called die
 * The specific snippet of code:
 *   				die "${FUNCNAME[1]}: unexpected portageq exit code: ${retval}"
 *
 * If you need support, post the output of `emerge --info '=app-admin/perl-cleaner-2.30::gentoo_prefix'`,
 * the complete build log and the output of `emerge -pqv '=app-admin/perl-cleaner-2.30::gentoo_prefix'`.
 * The complete build log is located at '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/temp/build.log'.
 * The ebuild environment file is located at '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/temp/environment'.
 * Working directory: '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/image'
 * S: '/Users/lightsing/Gentoo/var/tmp/portage/app-admin/perl-cleaner-2.30/work/perl-cleaner-2.30'
Comment 30 Fabian Groffen gentoo-dev 2022-08-04 15:25:25 UTC
yes, need to figure out how to get it to use python3.10 from the start
Comment 31 Fabian Groffen gentoo-dev 2022-08-04 15:27:48 UTC
oh crap, portage doesn't really handle a python upgrade
Comment 32 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-08-04 15:52:38 UTC
(In reply to Fabian Groffen from comment #31)
> oh crap, portage doesn't really handle a python upgrade

It does normally, this is a bug in the rewritten QA check. I'll have a look in a minute
Comment 33 Fabian Groffen gentoo-dev 2022-08-04 17:28:12 UTC
bootstrapping with python-3.10 is easier, I'm running that as we speak to confirm if it works
Comment 34 Askar Bektassov 2022-08-04 20:17:16 UTC
Changed 

- PYTHONMAJMIN=3.9   # keep this number in line with PV below for stage1,2
  bootstrap_python() {
-         PV=3.9.13

to

+ PYTHONMAJMIN=3.10   # keep this number in line with PV below for stage1,2
  bootstrap_python() {
+         PV=3.10.4 

Run bootstrap from scratch. After completing stage1, the script fails as follows.

ln: failed to create symbolic link '/Users/askarbektassov/Gentoo/tmp/usr/lib/python?.?/portage': No such file or directory
ln: failed to create symbolic link '/Users/askarbektassov/Gentoo/tmp/usr/lib/python?.?/_emerge': No such file or directory
* prefix-portage-3.0.30.1 successfully bootstrapped
* stage1 successfully finished
Traceback (most recent call last):
  File "/Users/askarbektassov/Gentoo/tmp/usr/bin/portageq", line 43, in <module>
    import portage
ModuleNotFoundError: No module named 'portage'
* Bootstrapping Gentoo prefixed portage installation using
* host:   arm64-apple-darwin21
* prefix: /Users/askarbektassov/Gentoo
* ready to bootstrap stage2_log
* Triggering Darwin with GCC toolchain
USE=-acl -berkdb -fortran -gdbm -git -libcxx -nls -pcre -python -qmanifest -qtegrity -readline -sanitize bootstrap clang internal-glib PKG=sys-devel/gnuconfig
Traceback (most recent call last):
  File "/Users/askarbektassov/Gentoo/tmp/usr/bin/emerge", line 40, in <module>
    import portage
ModuleNotFoundError: No module named 'portage'
Comment 35 Larry the Git Cow gentoo-dev 2022-08-04 23:49:45 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c310e44692ad91777c71394dda97cd1f33f72589

commit c310e44692ad91777c71394dda97cd1f33f72589
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-08-04 23:43:05 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-08-04 23:48:57 +0000

    metadata/install-qa-check.d: don't fail if has_version died
    
    During Python target migrations, if portage[-ipc], portageq can't
    be invoked as a binary (which has_version does behind the scenes w/ -ipc),
    so just call it w/ nonfatal, to avoid aborting the emerge and possibly
    hurting the system (imagine python-exec gets upgraded but portage
    wasn't yet).
    
    Fails like:
    ```
    
    ${EPREFIX}/Gentoo/usr/lib/python-exec/python3.9/portageq: this Python implementation (python3.9) is not supported by the script.
     * ERROR: app-admin/perl-cleaner-2.30::gentoo_prefix failed:
     *   has_version: unexpected portageq exit code: 127
     *
     * Call stack:
     *   misc-functions.sh, line 1334:  Called install_qa_check
     *   misc-functions.sh, line  138:  Called source 'install_symlink_html_docs'
     *        60python-pyc, line  118:  Called python_pyc_check
     *        60python-pyc, line   22:  Called has_version 'install_hooks'
     *    phase-helpers.sh, line  986:  Called ___best_version_and_has_version_common '${EPREFIX}/Gentoo/var/db/repos/gentoo/metadata/install-qa-check.d/60python-pyc'
     *    phase-helpers.sh, line  973:  Called die
     * The specific snippet of code:
     *                              die "${FUNCNAME[1]}: unexpected portageq exit code: ${retval}"
    ```
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Sam James <sam@gentoo.org>

 metadata/install-qa-check.d/60python-pyc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 36 Larry the Git Cow gentoo-dev 2022-08-05 06:35:54 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=fd1c9f9b6b838173e6f7df0cb8cc5114e55f6c56

commit fd1c9f9b6b838173e6f7df0cb8cc5114e55f6c56
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-05 06:32:34 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-05 06:32:34 +0000

    scripts/bootstrap-prefix: bump bootstrap python to 3.10 to match tree
    
    There is a problem which makes it hard to upgrade Python without
    transition in place (addressed by
    c310e44692ad91777c71394dda97cd1f33f72589?), so just use a bootstrap
    Python that's equal to the version installed by the tree to avoid this
    problem.
    
    While at it, drop arm64-macos hack, since it's no longer necessary.
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
Comment 37 Fabian Groffen gentoo-dev 2022-08-05 06:41:11 UTC
(In reply to Askar Bektassov from comment #34)
> Changed 
> 
> - PYTHONMAJMIN=3.9   # keep this number in line with PV below for stage1,2
>   bootstrap_python() {
> -         PV=3.9.13
> 
> to
> 
> + PYTHONMAJMIN=3.10   # keep this number in line with PV below for stage1,2
>   bootstrap_python() {
> +         PV=3.10.4 

This was good (I committed the same thing now, as I was testing yesterday)

> ln: failed to create symbolic link
> '/Users/askarbektassov/Gentoo/tmp/usr/lib/python?.?/portage': No such file
> or directory

This revealed a problem we had for a while, now solved for real (I hope).

Thanks
Comment 38 Fabian Groffen gentoo-dev 2022-08-05 10:40:13 UTC
this now gets stuck on groff:

ld: in section __TEXT,__text reloc 442: symbol index out of range file 'src/preproc/refer/refer-label.o' for architecture arm64
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:6438: refer] Error 1
make[1]: *** Waiting for unfinished jobs....

This can be worked around by forcing -O1 in CXXFLAGS.  I'm not yet sure what's the best solution for this.  On my laptop the same thing with -O2 works fine, while it is the same "M1".  So I guess there's still a subtle difference somewhere.  This MacMini is considerably newer than my laptop.
Comment 39 Askar Bektassov 2022-08-05 20:33:57 UTC
(In reply to Fabian Groffen from comment #38)
> this now gets stuck on groff:
> 
> ld: in section __TEXT,__text reloc 442: symbol index out of range file
> 'src/preproc/refer/refer-label.o' for architecture arm64
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:6438: refer] Error 1
> make[1]: *** Waiting for unfinished jobs....
> 
> This can be worked around by forcing -O1 in CXXFLAGS.  I'm not yet sure
> what's the best solution for this.  On my laptop the same thing with -O2
> works fine, while it is the same "M1".  So I guess there's still a subtle
> difference somewhere.  This MacMini is considerably newer than my laptop.

Same here, had to force CXXFLAGS to complete stage3

$ CXXFLAGS="-O1 -pipe" emerge groff

My machine:

askarbektassov@MacBook-Pro ~ % system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,3
      Chip: Apple M1 Pro
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 32 GB

Final note, MacOS is now pushing zsh as default option. This means that when I run bootstrap-prefix.sh startscript, it creates startprefix that does not start bash (with the accustomed theme). Once I am inside Prefix, I have to launch bash by hand (see below).

Entering Gentoo Prefix /Users/askarbektassov/Gentoo
MacBook-Pro% bash
askarbektassov@MacBook-Pro ~ $

Am I doing it wrong? Is there a more elegant solution (do I really need to add bash as a command to be executed upon launch)?
Comment 40 Fabian Groffen gentoo-dev 2022-08-06 08:14:14 UTC
this is by design, it takes over your shell, if you want that different, you can do something like 

% env SHELL=bash /path/to/startprefix

or run chsh

I think it's safest if you just hack the startprefix script to use the shell you want (just export SHELL=bash at the start of the script) or create a wrapper script to call startprefix in the way above.
Comment 41 Askar Bektassov 2022-08-06 09:49:07 UTC
Hey Fabien, thanks a lot, I have a fully working Gentoo instance now :)

As for the terminal, at the end I decided to start using zsh (I defeated my reluctance and started using it). For starters I installed oh-my-zsh from within macOS zsh (strangely I had some issues getting git to work from within Gentoo) and then turned on "gentoo" theme, already available.

Again, you are doing an impressive job. Thanks a lot!
Comment 42 Fabian Groffen gentoo-dev 2022-08-06 10:28:17 UTC
Cool, beware that other packages might need -O1 trick too.  I had it for vim for instance.  May be resolved in vim-9, but I'm not sure, as I think I switched to -O1 globally for the time being.  (Working Prefix is better than a wee bit faster one.)

That said, perhaps I should just make the bootstrap use -O1 on arm64-darwin, as I don't think we can use package.env.  Perhaps I could hack a static list in the profile's profile.bashrc to fix the issue (I believe it is a toolchain problem, not the package itself, so don't want to fix it in the ebuild.)
Comment 43 Larry the Git Cow gentoo-dev 2022-08-06 10:54:21 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ef78854564d1927cd8c44099287446049fe7521f

commit ef78854564d1927cd8c44099287446049fe7521f
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-06 10:52:25 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-06 10:54:18 +0000

    prefix/darwin/macos/arch/arm64: force -O1 where necessary as workaround
    
    The GCC toolchain seems somewhat funky for certain packages, producing
    objects that the (host) linker doesn't agree with, so for the time being
    force -O1 as workaround for these few packages.
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 .../prefix/darwin/macos/arch/arm64/profile.bashrc  | 26 +++++++++++++++-------
 1 file changed, 18 insertions(+), 8 deletions(-)
Comment 44 Larry the Git Cow gentoo-dev 2022-08-08 08:09:41 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=04d477f6f9a4a51e71d35d200c10ce47ca84c65b

commit 04d477f6f9a4a51e71d35d200c10ce47ca84c65b
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2022-08-08 08:08:40 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2022-08-08 08:08:40 +0000

    scripts/bootstrap-prefix: bump snapshot for arm64-macos profile fixes
    
    This should allow a bootstrap to succeed on arm64-macos, bypassing the
    groff build problem.
    
    Bug: https://bugs.gentoo.org/778014
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 scripts/bootstrap-prefix.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 45 Fabian Groffen gentoo-dev 2022-08-08 15:32:49 UTC
I just finished a successful bootstrap on arm64-macos.  Sorry for the long wait everyone!
Comment 46 Benda Xu gentoo-dev 2022-08-08 16:45:01 UTC
Fabian, great job! Yay
Comment 47 * 2022-08-29 03:51:26 UTC
Amazing job! Thanks so much! Fingers crossed that Ventura will be at least a bit easier.