Bug 780810

Summary:	sys-libs/musl-1.2.2-r2 Failed to emerge in a docker container
Product:	Gentoo Linux	Reporter:	Michel Ganguin <ganguin>
Component:	Current packages	Assignee:	Anthony Basile <blueness>
Status:	RESOLVED WORKSFORME
Severity:	normal	CC:	bertrand, jstein, lu_zero, sam, toolchain
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
Whiteboard:
Package list:		Runtime testing required:	---
Attachments:	emerge --info build.log on emerge musl build.log on any emerge invocation after musl 1.2.2 installed

Description Michel Ganguin 2021-04-07 11:11:52 UTC

Emerge of sys-libs/musl-1.2.2-r2 in a docker container fails with the message >>> Failed to execute postinst for sys-libs/musl-1.2.2-r2. The log contains multiple occurrences of `File not found: /usr/lib/portage/python3.8/ebuild-ipc`.

The ebuild-ipc runs `if [[ -x ...`, but the tested file is there, has the executable bits, but -x says it's not. It does not happen with musl 1.2.1. (When manually installing musl 1.2.2, the error happens for every emerge invocations.)

I tried disabling portage sanboxing, but same issue. I also tried to reproduce outside of docker, but it didn't fail (I tried in a chroot and in a systemd-nspawn). So it seems related to docker, but I couldn't find out why.

Reproducible: Always

Steps to Reproduce:
minimal Dockerfile:

```
# syntax = docker/dockerfile:1.2

FROM gentoo/stage3:amd64-musl-hardened
RUN --mount=from=gentoo/portage:latest,source=/var/db/repos/gentoo,target=/var/db/repos/gentoo,rw emerge -uv musl
CMD ["/bin/bash"]
```

`DOCKER_BUILDKIT=1 docker build .`


Actual Results:  
emerge fails with after the install phase:

```
#8 35.21 File not found: /usr/lib/portage/python3.8/ebuild-ipc
#8 35.21  * The ebuild phase 'postrm' has exited unexpectedly. This type of behavior
#8 35.21  * is known to be triggered by things such as failed variable assignments
#8 35.21  * (bug #190128) or bad substitution errors (bug #200313). Normally, before
#8 35.21  * exiting, bash should have displayed an error message above. If bash did
#8 35.21  * not produce an error message above, it's possible that the ebuild has
#8 35.21  * called `exit` when it should have called `die` instead. This behavior
#8 35.21  * may also be triggered by a corrupt bash binary or a hardware problem
#8 35.21  * such as memory or cpu malfunction. If the problem is not reproducible or
#8 35.21  * it appears to occur randomly, then it is likely to be triggered by a
#8 35.21  * hardware problem. If you suspect a hardware problem then you should try
#8 35.21  * some basic hardware diagnostics such as memtest. Please do not report
#8 35.21  * this as a bug unless it is consistently reproducible and you are sure
#8 35.21  * that your bash binary and hardware are functioning properly.
#8 35.27 File not found: /usr/lib/portage/python3.8/ebuild-ipc
#8 35.27  * The ebuild phase 'die_hooks' has exited unexpectedly. This type of
#8 35.27  * behavior is known to be triggered by things such as failed variable
#8 35.27  * assignments (bug #190128) or bad substitution errors (bug #200313).
#8 35.27  * Normally, before exiting, bash should have displayed an error message
#8 35.27  * above. If bash did not produce an error message above, it's possible
#8 35.27  * that the ebuild has called `exit` when it should have called `die`
#8 35.27  * instead. This behavior may also be triggered by a corrupt bash binary or
#8 35.27  * a hardware problem such as memory or cpu malfunction. If the problem is
#8 35.27  * not reproducible or it appears to occur randomly, then it is likely to
#8 35.27  * be triggered by a hardware problem. If you suspect a hardware problem
#8 35.27  * then you should try some basic hardware diagnostics such as memtest.
#8 35.27  * Please do not report this as a bug unless it is consistently
#8 35.27  * reproducible and you are sure that your bash binary and hardware are
#8 35.27  * functioning properly.
#8 35.27 !!! FAILED postrm: 1
#8 35.27  * The 'postrm' phase of the 'sys-libs/musl-1.2.1-r2' package has failed
#8 35.27  * with exit value 1.
#8 35.27  *
#8 35.27  * The problem occurred while executing the ebuild file named
#8 35.27  * 'musl-1.2.1-r2.ebuild' located in the '/var/db/pkg/sys-
#8 35.27  * libs/musl-1.2.1-r2' directory. If necessary, manually remove the
#8 35.27  * environment.bz2 file and/or the ebuild file located in that directory.
#8 35.27  *
#8 35.27  * Removal of the environment.bz2 file is preferred since it may allow the
#8 35.27  * removal phases to execute successfully. The ebuild will be sourced and
#8 35.27  * the eclasses from the current ebuild repository will be used when
#8 35.27  * necessary. Removal of the ebuild file will cause the pkg_prerm() and
#8 35.27  * pkg_postrm() removal phases to be skipped entirely.
#8 35.28 >>> Regenerating /etc/ld.so.cache...
#8 35.39 >>> Original instance of package unmerged safely.
#8 35.54 File not found: /usr/lib/portage/python3.8/ebuild-ipc
#8 35.55  * The ebuild phase 'postinst' has exited unexpectedly. This type of
#8 35.55  * behavior is known to be triggered by things such as failed variable
#8 35.55  * assignments (bug #190128) or bad substitution errors (bug #200313).
#8 35.55  * Normally, before exiting, bash should have displayed an error message
#8 35.55  * above. If bash did not produce an error message above, it's possible
#8 35.55  * that the ebuild has called `exit` when it should have called `die`
#8 35.55  * instead. This behavior may also be triggered by a corrupt bash binary or
#8 35.55  * a hardware problem such as memory or cpu malfunction. If the problem is
#8 35.55  * not reproducible or it appears to occur randomly, then it is likely to
#8 35.55  * be triggered by a hardware problem. If you suspect a hardware problem
#8 35.55  * then you should try some basic hardware diagnostics such as memtest.
#8 35.55  * Please do not report this as a bug unless it is consistently
#8 35.55  * reproducible and you are sure that your bash binary and hardware are
#8 35.55  * functioning properly.
#8 35.60 File not found: /usr/lib/portage/python3.8/ebuild-ipc
#8 35.61  * The ebuild phase 'die_hooks' has exited unexpectedly. This type of
#8 35.61  * behavior is known to be triggered by things such as failed variable
#8 35.61  * assignments (bug #190128) or bad substitution errors (bug #200313).
#8 35.61  * Normally, before exiting, bash should have displayed an error message
#8 35.61  * above. If bash did not produce an error message above, it's possible
#8 35.61  * that the ebuild has called `exit` when it should have called `die`
#8 35.61  * instead. This behavior may also be triggered by a corrupt bash binary or
#8 35.61  * a hardware problem such as memory or cpu malfunction. If the problem is
#8 35.61  * not reproducible or it appears to occur randomly, then it is likely to
#8 35.61  * be triggered by a hardware problem. If you suspect a hardware problem
#8 35.61  * then you should try some basic hardware diagnostics such as memtest.
#8 35.61  * Please do not report this as a bug unless it is consistently
#8 35.61  * reproducible and you are sure that your bash binary and hardware are
#8 35.61  * functioning properly.
#8 35.61  * FAILED postinst: 1
#8 35.61 >>> Regenerating /etc/ld.so.cache...
#8 35.78 File not found: /usr/lib/portage/python3.8/ebuild-ipc
#8 35.79  * The ebuild phase 'other' has exited unexpectedly. This type of behavior
#8 35.79  * is known to be triggered by things such as failed variable assignments
#8 35.79  * (bug #190128) or bad substitution errors (bug #200313). Normally, before
#8 35.79  * exiting, bash should have displayed an error message above. If bash did
#8 35.79  * not produce an error message above, it's possible that the ebuild has
#8 35.79  * called `exit` when it should have called `die` instead. This behavior
#8 35.79  * may also be triggered by a corrupt bash binary or a hardware problem
#8 35.79  * such as memory or cpu malfunction. If the problem is not reproducible or
#8 35.79  * it appears to occur randomly, then it is likely to be triggered by a
#8 35.79  * hardware problem. If you suspect a hardware problem then you should try
#8 35.79  * some basic hardware diagnostics such as memtest. Please do not report
#8 35.79  * this as a bug unless it is consistently reproducible and you are sure
#8 35.79  * that your bash binary and hardware are functioning properly.
#8 35.94
#8 35.94 >>> Recording sys-libs/musl in "world" favorites file...
#8 35.94
#8 35.94 >>> Failed to execute postinst for sys-libs/musl-1.2.2-r2
#8 35.94  * Messages for package sys-libs/musl-1.2.1-r2:
#8 35.94  * The ebuild phase 'prerm' has exited unexpectedly. This type of behavior
#8 35.94  * is known to be triggered by things such as failed variable assignments
#8 35.94  * (bug #190128) or bad substitution errors (bug #200313). Normally, before
#8 35.94  * exiting, bash should have displayed an error message above. If bash did
#8 35.94  * not produce an error message above, it's possible that the ebuild has
#8 35.94  * called `exit` when it should have called `die` instead. This behavior
#8 35.94  * may also be triggered by a corrupt bash binary or a hardware problem
#8 35.94  * such as memory or cpu malfunction. If the problem is not reproducible or
#8 35.94  * it appears to occur randomly, then it is likely to be triggered by a
#8 35.94  * hardware problem. If you suspect a hardware problem then you should try
#8 35.94  * some basic hardware diagnostics such as memtest. Please do not report
#8 35.94  * this as a bug unless it is consistently reproducible and you are sure
#8 35.94  * that your bash binary and hardware are functioning properly.
#8 35.94  * The ebuild phase 'postrm' has exited unexpectedly. This type of behavior
#8 35.94  * is known to be triggered by things such as failed variable assignments
#8 35.94  * (bug #190128) or bad substitution errors (bug #200313). Normally, before
#8 35.94  * exiting, bash should have displayed an error message above. If bash did
#8 35.94  * not produce an error message above, it's possible that the ebuild has
#8 35.94  * called `exit` when it should have called `die` instead. This behavior
#8 35.94  * may also be triggered by a corrupt bash binary or a hardware problem
#8 35.94  * such as memory or cpu malfunction. If the problem is not reproducible or
#8 35.94  * it appears to occur randomly, then it is likely to be triggered by a
#8 35.94  * hardware problem. If you suspect a hardware problem then you should try
#8 35.94  * some basic hardware diagnostics such as memtest. Please do not report
#8 35.94  * this as a bug unless it is consistently reproducible and you are sure
#8 35.94  * that your bash binary and hardware are functioning properly.
#8 35.94  * The 'postrm' phase of the 'sys-libs/musl-1.2.1-r2' package has failed
#8 35.94  * with exit value 1.
#8 35.94  *
#8 35.94  * The problem occurred while executing the ebuild file named
#8 35.94  * 'musl-1.2.1-r2.ebuild' located in the '/var/db/pkg/sys-
#8 35.94  * libs/musl-1.2.1-r2' directory. If necessary, manually remove the
#8 35.94  * environment.bz2 file and/or the ebuild file located in that directory.
#8 35.94  *
#8 35.94  * Removal of the environment.bz2 file is preferred since it may allow the
#8 35.94  * removal phases to execute successfully. The ebuild will be sourced and
#8 35.94  * the eclasses from the current ebuild repository will be used when
#8 35.94  * necessary. Removal of the ebuild file will cause the pkg_prerm() and
#8 35.94  * pkg_postrm() removal phases to be skipped entirely.
#8 35.94  * Messages for package sys-libs/musl-1.2.2-r2:
#8 35.94  * The ebuild phase 'postinst' has exited unexpectedly. This type of
#8 35.94  * behavior is known to be triggered by things such as failed variable
#8 35.94  * assignments (bug #190128) or bad substitution errors (bug #200313).
#8 35.94  * Normally, before exiting, bash should have displayed an error message
#8 35.94  * above. If bash did not produce an error message above, it's possible
#8 35.94  * that the ebuild has called `exit` when it should have called `die`
#8 35.94  * instead. This behavior may also be triggered by a corrupt bash binary or
#8 35.94  * a hardware problem such as memory or cpu malfunction. If the problem is
#8 35.94  * not reproducible or it appears to occur randomly, then it is likely to
#8 35.94  * be triggered by a hardware problem. If you suspect a hardware problem
#8 35.94  * then you should try some basic hardware diagnostics such as memtest.
#8 35.94  * Please do not report this as a bug unless it is consistently
#8 35.94  * reproducible and you are sure that your bash binary and hardware are
#8 35.94  * functioning properly.
#8 35.94  * FAILED postinst: 1
#8 35.94  * The ebuild phase 'other' has exited unexpectedly. This type of behavior
#8 35.94  * is known to be triggered by things such as failed variable assignments
#8 35.94  * (bug #190128) or bad substitution errors (bug #200313). Normally, before
#8 35.94  * exiting, bash should have displayed an error message above. If bash did
#8 35.94  * not produce an error message above, it's possible that the ebuild has
#8 35.94  * called `exit` when it should have called `die` instead. This behavior
#8 35.94  * may also be triggered by a corrupt bash binary or a hardware problem
#8 35.94  * such as memory or cpu malfunction. If the problem is not reproducible or
#8 35.94  * it appears to occur randomly, then it is likely to be triggered by a
#8 35.94  * hardware problem. If you suspect a hardware problem then you should try
#8 35.94  * some basic hardware diagnostics such as memtest. Please do not report
#8 35.94  * this as a bug unless it is consistently reproducible and you are sure
#8 35.94  * that your bash binary and hardware are functioning properly.
#8 35.94
#8 35.94
#8 35.94
#8 35.94
#8 35.95
#8 35.95  * GNU info directory index is up-to-date.
#8 35.97
#8 35.97  * IMPORTANT: 6 news items need reading for repository 'gentoo'.
#8 35.97  * Use eselect news read to view new items.
#8 35.97
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c emerge -uv musl]: runc did not terminate sucessfully
```

Expected Results:  
upgrade musl.

Comment 1 Michel Ganguin 2021-04-07 11:19:24 UTC

Note that doing interactively does the same:

docker create -v /var/db/repos/gentoo --name portage gentoo/portage:latest /bin/true

docker run --volumes-from portage -it --rm  gentoo/stage3:amd64-musl-hardened

emerge -v musl

Comment 2 Sergei Trofimovich (RETIRED) gentoo-dev

2021-04-16 21:26:41 UTC

Can you attach example build.log as it's created by portage? Also can you add `emerge --info` from that system?

Chances are that you need to disable namespaces related features.

Comment 3 Michel Ganguin 2021-04-16 23:51:37 UTC

Created attachment 700275 [details]
emerge --info

As I already mentioned, it is unmodified stage3 from docker hub

gentoo/stage3:amd64-musl-hardened

Comment 4 Michel Ganguin 2021-04-16 23:57:42 UTC

Created attachment 700278 [details]
build.log on emerge musl

build.log (/var/tmp/portage/._unmerge_/sys-libs/musl-1.2.1-r2/temp/build.log) on emerge musl (the install succeeds, fails in prerm and postinst phase)

Comment 5 Michel Ganguin 2021-04-16 23:58:37 UTC

Created attachment 700281 [details]
build.log on any emerge invocation after musl 1.2.2 installed

Comment 6 Michel Ganguin 2021-04-17 14:36:40 UTC

I tried with FEATURES="-sandbox", FEATURES="-ipc-sandbox" and both FEATURES="-sandbox -ipc-sandbox". Same issue.

Comment 7 Sergei Trofimovich (RETIRED) gentoo-dev

2021-06-11 08:10:12 UTC

(In reply to Michel Ganguin from comment #1)
> Note that doing interactively does the same:
> 
> docker create -v /var/db/repos/gentoo --name portage gentoo/portage:latest
> /bin/true
> 
> docker run --volumes-from portage -it --rm  gentoo/stage3:amd64-musl-hardened
> 
> emerge -v musl

That works for me as is without failures.

build.log does not give more insight. Looks like something very basic is broken. You might need to debug it yourself. Try:

# emerge -v musl --debug

Maybe it will be clearer how far into setup it got.

Comment 8 Michel Ganguin 2021-06-12 00:53:49 UTC

I cannot reproduce as well, I even tried older tags (gentoo/stage3:amd64-musl-hardened-20210407).

But my docker got updated in between, today I have:

app-emulation/containerd-1.4.4
app-emulation/docker-20.10.6-r1

Back in April I had:

app-emulation/containerd-1.3.9
app-emulation/docker-19.03.15

So it seems it was a docker issue?!

Comment 9 Sergei Trofimovich (RETIRED) gentoo-dev

2021-06-12 08:41:07 UTC

Might also be an older sandbox (or other) bug that we fixed by accident.

Comment 10 Sergei Trofimovich (RETIRED) gentoo-dev

2021-06-12 08:42:12 UTC

(In reply to Sergei Trofimovich from comment #9)
> Might also be an older sandbox (or other) bug that we fixed by accident.

Ah, you just mentioned that you tried the same image. Docker version sounds very plausible (or changed set of kernel features if you happened to change that).