Created attachment 691206 [details] build.log Running the test suite of dev-util/libabigail-1.8.2 seems to fail with at least sys-apps/sandbox-2.20 and sys-apps/sandbox-2.21. Here's the failure of libabigail-1.8.2: >make check-TESTS >PASS: runtestslowselfcompare.sh >/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2/build-aux/test-driver: line 109: 143 Segmentation fault (core dumped) "$@" > $log_file 2>&1 >FAIL: runtestdiffsuppr >/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2/build-aux/test-driver: line 109: 146 Segmentation fault (core dumped) "$@" > $log_file 2>&1 >FAIL: runtestdiffpkg >[...] Same thing outside of Portage: >/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests $ sandbox make check >Making check in data >make[1]: Entering directory '/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests/data' >make[1]: Nothing to be done for 'check'. >make[1]: Leaving directory '/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests/data' >make[1]: Entering directory '/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests' >make check-TESTS >make[2]: Entering directory '/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests' >make[3]: Entering directory '/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2_build/tests' >PASS: runtestslowselfcompare.sh >/var/tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2/build-aux/test-driver: line 109: 1970645 Segmentation fault (core dumped) "$@" > $log_file 2>&1 >FAIL: runtestdiffsuppr Let's try running a failing test individually: >$ sandbox ./runtestdiffsuppr >Sandboxed process killed by signal: Segmentation fault >Segmentation fault (core dumped) How about serially? >$ sandbox ./runtestdiffsuppr --no-parallel >?notice: Found incorrectly calculated array sizes in XML - this is benign. >Older versions of libabigail miscalculated multidimensional array sizes. >notice: Found incorrectly calculated array sizes in XML - this is benign. >Older versions of libabigail miscalculated multidimensional array sizes. >notice: Found incorrectly calculated array sizes in XML - this is benign. >Older versions of libabigail miscalculated multidimensional array sizes. Note: if tests succeed, you'll need to manually delete results to stop them autopassing again, it seems: > $ rm -rf *.out *.trs *.log output gdb didn't feel particularly insightful: >(gdb) r >Starting program: /usr/bin/sandbox ./runtestdiffsuppr >[Detaching after fork from child process 2013300] >Sandboxed process killed by signal: Segmentation fault > >Program received signal SIGSEGV, Segmentation fault. >0x00007ffff7e24e37 in kill () at ../sysdeps/unix/syscall-template.S:120 >120 ../sysdeps/unix/syscall-template.S: No such file or directory. >(gdb) bt >#0 0x00007ffff7e24e37 in kill () at ../sysdeps/unix/syscall-template.S:120 >#1 0x0000555555558429 in main (argc=<optimized out>, argv=<optimized out>) > at /usr/src/debug/sys-apps/sandbox-2.21/sandbox-2.21/src/sandbox.c:346 >(gdb) I started investigating this with libabigail upstream who made some suggestions. Some of the failures seemed related to PYTHONDONTWRITEBYTECODE=1 (probably just when outside of Portage). I ended up getting failures showing the specific command failing being when calling system() for 'diff' on results which was nonsense. I don't recall right now how I managed to get that slightly more useful backtrace. I could get it to consistently show diff being the failing command, however. The earlier version of libabigail in tree (dev-util/libabigail-1.6) seems to pass tests. Following a conversation with slyfox in #gentoo-toolchain: > <@slyfox> if it's a multithreaded environment it's might be sandbox changing environment similar to how uwsgi was affected: https://bugs.gentoo.org/680058#c10 > <+sam_> i'm wondering if this is related to the libabigail issue > <+sam_> or rather, libabigail issue is related to this > <+sam_> slyfox: I see a similar issue when bumping libabigail, I end up hitting an error when it calls diff > <@slyfox> to verify you can try reverting https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=f3e51a930312422cc78b693a247b7c5704ac90a2 > <@slyfox> (it trades one set of environment poisoning bugs for another) > <@slyfox> if problems disappear it is probably the smilar to uwsgi failure > <@slyfox> but sandbox@ should know better. i suggest filing a bug against them. I tried reverting the linked commit on top of sandbox-2.21 and this allowed tests to pass again.
Crashed locally as well. The crash happens right at environ parsing: #0 0x00007f0e0d10e392 in sb_new_envp () at libsandbox.c:1184 1184 libsandbox.c: No such file or directory. [Current thread is 1 (Thread 0x7f0e0b433640 (LWP 1699029))] (gdb) bt #0 0x00007f0e0d10e392 in sb_new_envp () at libsandbox.c:1184 #1 0x00007f0e0d112745 in system_DEFAULT () at wrapper-funcs/__wrapper_exec.c:315 #2 0x000055ece6570b52 in test_task::perform (this=0x55ece72978b0) at /usr/lib/gcc/x86_64-pc-linux-gnu/11.0.1/include/g++-v11/bits/basic_string.h:186 #3 0x00007f0e0d05fce5 in abigail::workers::worker::wait_to_execute_a_task (p=0x55ece728f380) at /tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2/src/abg-workers.cc:415 #4 0x00007f0e0c8f7cae in start_thread (arg=0x7f0e0b433640) at pthread_create.c:473 #5 0x00007f0e0ca0eb2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 The test roughly does: for_each_test(){ spawn_thread([]{ system("cmd"); wait(); } } system() does guarantee reasonable behaviour in multithreaded programs. sandbox mutates global 'environ' to adjust sandbox injection when needed. https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=f3e51a930312422cc78b693a247b7c5704ac90a2 has two semantic changes that might be relevant here as: 1. It slightly widens (or narrows, depends on how you look at it) the gap of race window when 'environ' is traversed. 2. It always (I'll double-check a bit later) relocates 'environ' in memory if there are any changes to make in environ. Previously it had a chance of inplace mutation. 3. environ is always restored back after system() return. We might get away with it if we better understand why environ has to be updated. Maybe we change it for no reason (unlikely). Worst case we'll need to roll back environment change. It should not be too bad as we recently added more conservative vfork() handling.
Created attachment 691254 [details, diff] 0001-libsandbox-try-harder-not-to-regenerate-environment.patch Ended up being simple bug of incorrect detection of already set environment of `SANDBOX_DENY`. Try 0001-libsandbox-try-harder-not-to-regenerate-environment.patch. It fixed libabigail tests for me.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=b321cd403c653d5bac54b5ec8341bc631fe3331e commit b321cd403c653d5bac54b5ec8341bc631fe3331e Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2021-03-13 22:35:53 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2021-03-13 22:53:39 +0000 libsandbox: try harder not to regenerate environment In bug #775842 Sam noticed sandbox crash on libabigail-1.8.2 testsuite: #0 0x00007f0e0d10e392 in sb_new_envp () at libsandbox.c:1184 #1 0x00007f0e0d112745 in system_DEFAULT () at wrapper-funcs/__wrapper_exec.c:315 #2 0x000055ece6570b52 in test_task::perform (this=0x55ece72978b0) at /usr/lib/gcc/x86_64-pc-linux-gnu/11.0.1/include/g++-v11/bits/basic_string.h:186 #3 0x00007f0e0d05fce5 in abigail::workers::worker::wait_to_execute_a_task (p=0x55ece728f380) at /tmp/portage/dev-util/libabigail-1.8.2/work/libabigail-1.8.2/src/abg-workers.cc:415 #4 0x00007f0e0c8f7cae in start_thread (arg=0x7f0e0b433640) at pthread_create.c:473 #5 0x00007f0e0ca0eb2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 The test roughly does call system() from spawned parallel threads: for_each_test(){ spawn_thread([]{ system("cmd"); wait(); } } Sandbox has to inject sandbox-specific environment variables where they don't exist. This is fundamentally racy in multithreaded environment: for_each_test(){ spawn_thread([]{ environ = modified_env; system("cmd"); wait(); environ = orig_env; } } Most of the time environment does not have to change after initial environment injection. f3e51a9303124 ("exec*() wrappers: never mutate 'environ' of host process") exposed a bug in sandbox's logic of checking existing environment: unset variables like `SANDBOX_DENY` The change treats unset/expected-unset variables as non deviating. Reported-by: Sam James Bug: https://bugs.gentoo.org/775842 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> libsandbox/libsandbox.c | 9 +++++++++ 1 file changed, 9 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3851698e1f344071ec8f190ffcdd739821c4a07d commit 3851698e1f344071ec8f190ffcdd739821c4a07d Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2021-03-27 11:28:47 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2021-03-27 11:44:56 +0000 sys-apps/sandbox: bump up to 2.22 Single new change: - libsandbox: try harder not to regenerate environment Reported-by: Sam James Closes: https://bugs.gentoo.org/775842 Package-Manager: Portage-3.0.17, Repoman-3.0.2 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> sys-apps/sandbox/Manifest | 1 + sys-apps/sandbox/sandbox-2.22.ebuild | 54 ++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+)
