Observed since 2 daas at the tinderbox (pstree will be attached).
Created attachment 560826 [details] 29.20190113-040328
Created attachment 560838 [details] pstree-a.txt
FWIW this issue/regression of https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=f3e51a930312422cc78b693a247b7c5704ac90a2 seems to be package specific.
Do I understand correctly unexpected threads here are only 'erl_child_setup' and 'bash' ones and those are zombies? Or something else as well? The patch you linked (unless bugs) should not change behaviour of thread creation/shutdown. SIGCHLD handling might have changed by 'pid-ns-init' if for whatever reason 'pid-ns-init' disables SIGCHLD and sandbox inherits it. If you can easily reproduce it can you also try FEATURES=-pid-sandbox to see if it helps? AFAIU known reproducers are: dev-java/openjdk net-analyzer/tsung
(In reply to Sergei Trofimovich from comment #4) IMO they are not zombies, at least they are reaped at the end of the emerge process, but they do fill up quickly the process id table FEATURES=-pid-sandbox helped, reproducer was net-analyzer/tsung
(In reply to Toralf Förster from comment #5) > (In reply to Sergei Trofimovich from comment #4) > IMO they are not zombies, at least they are reaped at the end of the emerge > process, but they do fill up quickly the process id table > > FEATURES=-pid-sandbox helped, reproducer was net-analyzer/tsung Thanks for the info! I reproduced one locally and those are zombies at least for me: ... 21637 root 20 0 13760 7760 4736 S 0.0 0.0 0:00.02 └─ /usr/bin/python3.6m /usr/lib/portage/python3.6/pid-ns-init 11683 21638 portage 20 0 2384 1704 1552 S 0.0 0.0 0:00.00 └─ [dev-lang/erlang-21.1.1] sandbox /usr/lib/portage/python3.6/ebuild.sh compile 32507 portage 20 0 0 0 0 Z 0.0 0.0 0:00.00 ├─ erl_child_setup 1024 32465 portage 20 0 0 0 0 Z 0.0 0.0 0:00.00 ├─ erl_child_setup 1024 32438 portage 20 0 0 0 0 Z 0.0 0.0 0:00.00 ├─ erl_child_setup 1024 32436 portage 20 0 0 0 0 Z 0.0 0.0 0:00.00 ├─ erl_child_setup 1024 ... Thus I suspect something in reparenting handling is way off in pid-sandbox case. It's a bit unclear what is pid-1 in new pid namespace. It ought to be a pid-ns-init but children are reparented to sandbox.
sandbox is really pid=1. I've checked it by tweaking the ebuild slightly: --- a/net-analyzer/tsung/tsung-1.7.0.ebuild +++ b/net-analyzer/tsung/tsung-1.7.0.ebuild @@ -33,2 +33,4 @@ src_compile() { emake || die "Failed building" + pstree -a -p -s -S + die "enough" } The output is: sandbox,1 /usr/lib/portage/python3.6/ebuild.sh compile ├─ebuild.sh,4 /usr/lib/portage/python3.6/ebuild.sh compile │ └─ebuild.sh,21 /usr/lib/portage/python3.6/ebuild.sh compile │ └─pstree,4738 -a -p -s -S ├─(erl_child_setup,96) ├─(erl_child_setup,150)
FWIW the effect is much higher if I start the tinderox chroot with nice (due to the longer emerge time). The # of (zombie) processes is growing constantly with each second.
Without looking at the code (sorry) my guess is that portage calls exec() into 'pid-ns-init' without fork()ing a new process after unshare(CLONE_NEWPID). And that causes only next fork()ed process to ender new namespace. From 'man unshare': """ CLONE_NEWPID (since Linux 3.8) This flag has the same effect as the clone(2) CLONE_NEWPID flag. Unshare the PID namespace, so that the calling process has a new PID namespace for its children which is not shared with any previously existing process. The calling process is not moved into the new namespace. ... """
Here is our unshare()/execv() same-process sequence: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/process.py#n552
According to the pid_namespaces(7) man page, we need to arrange it so that pid-ns-init is the first process forked and execed immediately after the unshare call: > The namespace init process > > The first process created in a new namespace (i.e., the > process created using clone(2) with the CLONE_NEWPID flag, > or the first child created by a process after a call to > unshare(2) using the CLONE_NEWPID flag) has the PID 1, > and is the "init" process for the namespace (see init(1)). > A child process that is orphaned within the namespace > will be reparented to this process rather than init(1) > (unless one of the ancestors of the child in the same PID > names‐ pace employed the prctl(2) PR_SET_CHILD_SUBREAPER > command to mark itself as the reaper of orphaned descendant > pro‐ cesses).
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/d11a5aaa5aaa99f695f25bb093e10ce1 https://github.com/gentoo/portage/pull/398
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4c7debeccab9d1ea546ffbd6cbb9ff352aba8f63 commit 4c7debeccab9d1ea546ffbd6cbb9ff352aba8f63 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-01-15 06:13:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-01-15 06:22:23 +0000 sys-apps/portage: version bump to 2.3.56 #675284 restore canonicalize func #675312 pid-sandbox: execute pid-ns-init as pid 1 Bug: https://bugs.gentoo.org/670484 Bug: https://bugs.gentoo.org/671498 Bug: https://bugs.gentoo.org/675312 Package-Manager: Portage-2.3.56, Repoman-2.3.12 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.56.ebuild | 271 +++++++++++++++++++++++++++++++++ 2 files changed, 272 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=fb406579b1d13c1ba23b28e0bb794c22878a58c0 commit fb406579b1d13c1ba23b28e0bb794c22878a58c0 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-01-13 23:06:35 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-01-15 05:45:55 +0000 pid-sandbox: execute pid-ns-init as pid 1 (bug 675312) Execute pid-ns-init as the first fork after unshare, as required for it to have pid 1 and become the default reaper of orphaned descendant processes. In _exec, exec a separate pid-ns-init process to behave as a supervisor which will forward signals to init and forward exit status to the parent process. Fixes: a75d5546e3a4 ("Introduce a tiny init replacement for inside pid namespace") Bug: https://bugs.gentoo.org/675312 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> bin/pid-ns-init | 44 ++++++++++++++++++++++++++++++++++++++++---- lib/portage/process.py | 25 +++++++++++++++++++------ 2 files changed, 59 insertions(+), 10 deletions(-)
