Bootstrap process uses host software to compile basic tools in stage1. That's inevitable, it needs something to start with. But unfortunately bootstrap continues to use host software even after it compiled its own. It leads to harder debugging, because one can't be sure which tools were used in which situation and how exactly PATH affected bootstrapping process. My experience with that: I had a lot of stuff in my PATH and bootstrap was successfull on MacOS Ventura 13.3.1 with M1. At the end I got a warning about PATH modifications in my profile and that it may affect installed prefix. I figured Anaconda python created .profile and installed itself in it. After removing Anaconda from system and deleing .profile it became impossible for me to bootstrap even stage1, because of syntax errors between GNU and BSD utils like sed, find, etc. Anaconda had GNU tools and it seems that bootstrap-prefix.sh used them, instead of system ones. I suggest to use tools from prefix as soon as script compiled them. For example. Script downloaded and compiled make? Use it to build all packaged. Downloaded and compiled sed? Script should use it to change stuff. I'll show examples of errors that I get on MacOS and how they can be avoided in the comments below.
Created attachment 859978 [details] Full bootstrap log with sed error
(In reply to Andrey Aleksandrov from comment #1) > Created attachment 859978 [details] > Full bootstrap log with sed error This is how bootstrap-prefix.sh was executed: USE_CPU_CORES="2" LATEST_TREE_YES=1 ACCEPT_KEYWORDS="~x64-macos" GENTOO_MIRRORS="https://mirror.yandex.ru/gentoo-distfiles/" ~/Developer/prefix/scripts/bootstrap-prefix.sh "/Users/andreyaleksandrov/Developer/x64Gentoo" stage1 And it failed with sed syntax error * Unpacking prefix-portage-3.0.30.1 sed: -I or -i may not be used with stdin But at the moment script unpacks prefix-portage it already compiled sed and script could use it like this: diff --git a/scripts/bootstrap-prefix.sh b/scripts/bootstrap-prefix.sh index c032f7424a..199c46690e 100755 --- a/scripts/bootstrap-prefix.sh +++ b/scripts/bootstrap-prefix.sh @@ -673,13 +673,13 @@ bootstrap_portage() { fix_config_sub # disable ipc - sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ + ${ROOT}/tmp/bin/sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ -i lib/_emerge/AbstractEbuildProcess.py || \ return 1 # host-provided wget may lack certificates, stage1 wget is without ssl [[ $(wget -h) == *"--no-check-certificate"* ]] && - sed -e '/wget/s/ --passive-ftp /&--no-check-certificate /' -i cnf/make.globals + ${ROOT}/tmp/bin/sed -e '/wget/s/ --passive-ftp /&--no-check-certificate /' -i cnf/make.globals # Portage checks for valid shebangs. These may (xz-utils) originate # in CONFIG_SHELL (AIX), which originates in PORTAGE_BASH then.
With sed patch applied prefix-portage will start configuring and fail because where is no python. * Compiling prefix-portage-3.0.30.1 ./configure --host=arm64-apple-darwin22 --prefix=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr --mandir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share/man --infodir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share/info --datadir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share --sysconfdir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/etc --localstatedir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/var/lib --build=arm64-apple-darwin22 --with-offset-prefix=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp --with-portage-user=andreyaleksandrov --with-portage-group=staff --with-extra-path=/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/bin:/opt/homebrew/sbin checking build system type... aarch64-apple-darwin22 ... ... ... checking for python... no configure: error: no python found in your path MacOS has python3 in PATH, but no "python". But it doesn't matter, because at this point script already compiled python and we can use it. --- a/scripts/bootstrap-prefix.sh +++ b/scripts/bootstrap-prefix.sh @@ -690,6 +690,7 @@ bootstrap_portage() { [[ -x ${ROOT}/tmp/bin/bash ]] || ln -s "${BASH}" "${ROOT}"/tmp/bin/bash || return 1 [[ -x ${ROOT}/tmp/bin/sh ]] || ln -s bash "${ROOT}"/tmp/bin/sh || return 1 export PORTAGE_BASH="${ROOT}"/tmp/bin/bash + export PREFIX_PORTAGE_PYTHON=${ROOT}/tmp/bin/python einfo "Compiling ${A%.tar.*}" econf \ Configure process of prefix process also would raise errors like this: ``` checking for sed... /usr/bin/sed checking whether /usr/bin/sed is GNU sed... configure: error: no checking for find... /usr/bin/find checking whether /usr/bin/find is GNU find... configure: error: no checking for xargs... /usr/bin/xargs checking whether /usr/bin/xargs is GNU xargs... configure: error: no ``` And install erros, because BSD install in MacOS doesn't have -t option, but install at ${ROOT}/tmp/bin/install does. ``` /usr/bin/install -c -m 644 -m 755 -o andreyaleksandrov -g staff -t /Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/lib/portage/bin/./preinst-qa-check.d ././preinst-qa-check.d/50xdg-utils /usr/bin/install: illegal option -- t ```
Created attachment 859979 [details] Full bootstrap log til python error
Created attachment 859980 [details, diff] Final patch I made before creating this bug
Were this changes useful or this way of working is out of scope of prefix project and there is no point to spend more time on this?
I personally think there's value in this but I defer to grobian here. The more tools we use that we just-built, the more predictable their behaviour is, and the easier it is for us to debug them.
Yes this is very useful, except that tools are not bootstrapped if they are not necessary. So under linux I suppose this would cause bootstrap_portage to break, because sed is not compiled. I think we should check and see if we can setup PATH to first consider ROOT/tmp/usr/bin before the host paths.
Created attachment 860170 [details, diff] Patch to run part of interactive before every bootstrap_* Gave it another try, and it's not as bad as I thought. bootstrap-prefix.sh already modifies PATH and puts ROOT/tmp/usr/bin before everything else. And it works! But it does it only in interactive/noninteractive modes :) So I got this issue only because I execute the script with "stage1" argument. I tried to extract all PATH modifications from "bootstrap_interactive" but it looked messy. More functions and more login inside "bootstrap_interactive" to handle functions output. The simpliest fix I found is to always run "bootstrap_interactive" in non-interactive mode before any "bootstrap_<action>". But not fully, only the part that setups ENVs and checks installed tools.
this is a bit of a kludge, but I think this is indeed the best thing to make it work as expected at this point
Forgot commit ref, but I merged this as 7904f41ac9, thanks!
