Bug 904242 – bootstrap-prefix.sh in stage 1 should use tools from prefix as much as possible

• Product: Gentoo/Alt
• Status: RESOLVED FIXED

[Description Andrey Aleksandrov 2023-04-12 21:52:16 UTC]

Bootstrap process uses host software to compile basic tools in stage1. That's inevitable, it needs something to start with. But unfortunately bootstrap continues to use host software even after it compiled its own.

It leads to harder debugging, because one can't be sure which tools were used in which situation and how exactly PATH affected bootstrapping process.

My experience with that: I had a lot of stuff in my PATH and bootstrap was successfull on MacOS Ventura 13.3.1 with M1. At the end I got a warning about PATH modifications in my profile and that it may affect installed prefix. I figured Anaconda python created .profile and installed itself in it. After removing Anaconda from system and deleing .profile it became impossible for me to bootstrap even stage1, because of syntax errors between GNU and BSD utils like sed, find, etc. Anaconda had GNU tools and it seems that bootstrap-prefix.sh used them, instead of system ones.

I suggest to use tools from prefix as soon as script compiled them. For example. Script downloaded and compiled make? Use it to build all packaged. Downloaded and compiled sed? Script should use it to change stuff.

I'll show examples of errors that I get on MacOS and how they can be avoided in the comments below.

[Comment 1 Andrey Aleksandrov 2023-04-12 21:59:35 UTC]

Created attachment 859978 [details]
Full bootstrap log with sed error

[Comment 2 Andrey Aleksandrov 2023-04-12 22:05:30 UTC]

(In reply to Andrey Aleksandrov from comment #1)
> Created attachment 859978 [details]
> Full bootstrap log with sed error

This is how bootstrap-prefix.sh was executed:
USE_CPU_CORES="2"
LATEST_TREE_YES=1
ACCEPT_KEYWORDS="~x64-macos"
GENTOO_MIRRORS="https://mirror.yandex.ru/gentoo-distfiles/"

~/Developer/prefix/scripts/bootstrap-prefix.sh "/Users/andreyaleksandrov/Developer/x64Gentoo" stage1

And it failed with sed syntax error
* Unpacking prefix-portage-3.0.30.1
sed: -I or -i may not be used with stdin

But at the moment script unpacks prefix-portage it already compiled sed and script could use it like this:

diff --git a/scripts/bootstrap-prefix.sh b/scripts/bootstrap-prefix.sh
index c032f7424a..199c46690e 100755
--- a/scripts/bootstrap-prefix.sh
+++ b/scripts/bootstrap-prefix.sh
@@ -673,13 +673,13 @@ bootstrap_portage() {
 	fix_config_sub
 
 	# disable ipc
-	sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
+	${ROOT}/tmp/bin/sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
 		-i lib/_emerge/AbstractEbuildProcess.py || \
 		return 1
 
 	# host-provided wget may lack certificates, stage1 wget is without ssl
 	[[ $(wget -h) == *"--no-check-certificate"* ]] &&
-	sed -e '/wget/s/ --passive-ftp /&--no-check-certificate /' -i cnf/make.globals
+	${ROOT}/tmp/bin/sed -e '/wget/s/ --passive-ftp /&--no-check-certificate /' -i cnf/make.globals
 
 	# Portage checks for valid shebangs. These may (xz-utils) originate
 	# in CONFIG_SHELL (AIX), which originates in PORTAGE_BASH then.

[Comment 3 Andrey Aleksandrov 2023-04-12 22:42:13 UTC]

With sed patch applied prefix-portage will start configuring and fail because where is no python.

* Compiling prefix-portage-3.0.30.1
./configure --host=arm64-apple-darwin22 --prefix=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr --mandir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share/man --infodir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share/info --datadir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/share --sysconfdir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/etc --localstatedir=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp/var/lib --build=arm64-apple-darwin22 --with-offset-prefix=/Users/andreyaleksandrov/Developer/x64Gentoo/tmp --with-portage-user=andreyaleksandrov --with-portage-group=staff --with-extra-path=/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/bin:/opt/homebrew/sbin
checking build system type... aarch64-apple-darwin22
...
...
...
checking for python... no
configure: error: no python found in your path


MacOS has python3 in PATH, but no "python". But it doesn't matter, because at this point script already compiled python and we can use it.
--- a/scripts/bootstrap-prefix.sh
+++ b/scripts/bootstrap-prefix.sh
@@ -690,6 +690,7 @@ bootstrap_portage() {
        [[ -x ${ROOT}/tmp/bin/bash ]] || ln -s "${BASH}" "${ROOT}"/tmp/bin/bash || return 1
        [[ -x ${ROOT}/tmp/bin/sh ]] || ln -s bash "${ROOT}"/tmp/bin/sh || return 1
        export PORTAGE_BASH="${ROOT}"/tmp/bin/bash
+       export PREFIX_PORTAGE_PYTHON=${ROOT}/tmp/bin/python

        einfo "Compiling ${A%.tar.*}"
        econf \

Configure process of prefix process also would raise errors like this:
```
checking for sed... /usr/bin/sed
checking whether /usr/bin/sed is GNU sed... configure: error: no
checking for find... /usr/bin/find
checking whether /usr/bin/find is GNU find... configure: error: no
checking for xargs... /usr/bin/xargs
checking whether /usr/bin/xargs is GNU xargs... configure: error: no
```

And install erros, because BSD install in MacOS doesn't have -t option, but install at ${ROOT}/tmp/bin/install does.
```
/usr/bin/install -c -m 644 -m 755 -o andreyaleksandrov -g staff -t /Users/andreyaleksandrov/Developer/x64Gentoo/tmp/usr/lib/portage/bin/./preinst-qa-check.d
 ././preinst-qa-check.d/50xdg-utils
/usr/bin/install: illegal option -- t
```

[Comment 4 Andrey Aleksandrov 2023-04-12 22:43:56 UTC]

Created attachment 859979 [details]
Full bootstrap log til python error

[Comment 5 Andrey Aleksandrov 2023-04-12 22:46:52 UTC]

Created attachment 859980 [details, diff]
Final patch I made before creating this bug

[Comment 6 Andrey Aleksandrov 2023-04-12 22:52:04 UTC]

Were this changes useful or this way of working is out of scope of prefix project and there is no point to spend more time on this?

[Comment 7 Sam James 2023-04-13 02:31:02 UTC]

I personally think there's value in this but I defer to grobian here.

The more tools we use that we just-built, the more predictable their behaviour is, and the easier it is for us to debug them.

[Comment 8 Fabian Groffen 2023-04-13 06:16:45 UTC]

Yes this is very useful, except that tools are not bootstrapped if they are not necessary.  So under linux I suppose this would cause bootstrap_portage to break, because sed is not compiled.

I think we should check and see if we can setup PATH to first consider ROOT/tmp/usr/bin before the host paths.

[Comment 9 Andrey Aleksandrov 2023-04-15 19:37:32 UTC]

Created attachment 860170 [details, diff]
Patch to run part of interactive before every bootstrap_*

Gave it another try, and it's not as bad as I thought. bootstrap-prefix.sh already modifies PATH and puts ROOT/tmp/usr/bin before everything else. And it works! But it does it only in interactive/noninteractive modes :) So I got this issue only because I execute the script with "stage1" argument.

I tried to extract all PATH modifications from "bootstrap_interactive" but it looked messy. More functions and more login inside "bootstrap_interactive" to handle functions output.

The simpliest fix I found is to always run "bootstrap_interactive" in non-interactive mode before any "bootstrap_<action>". But not fully, only the part that setups ENVs and checks installed tools.

[Comment 10 Fabian Groffen 2023-04-17 15:38:31 UTC]

this is a bit of a kludge, but I think this is indeed the best thing to make it work as expected at this point

[Comment 11 Fabian Groffen 2023-04-25 15:26:02 UTC]

Forgot commit ref, but I merged this as 7904f41ac9, thanks!