Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 816396 - sys-apps/sandbox: rust's direct use of clone3 syscall leads to internal sandbox deadlocks
Summary: sys-apps/sandbox: rust's direct use of clone3 syscall leads to internal sandb...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Sandbox (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Sandbox Maintainers
URL: https://github.com/rust-lang/rust/iss...
Whiteboard:
Keywords:
Depends on:
Blocks: glibc-2.34
  Show dependency tree
 
Reported: 2021-10-05 09:45 UTC by 12101111
Modified: 2022-02-27 21:12 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description 12101111 2021-10-05 09:45:07 UTC
Rust 1.56 change the implement of Command::spawn to clone3 syscall on Linux (https://github.com/rust-lang/rust/pull/81825)

When run inside sandbox, cargo will hang after compile few dependence


Reproducible: Always

Steps to Reproduce:
1. Write ebuild for dev-lang/rust-1.56_beta or dev-lang/rust-bin-1.56_beta and install it
2. select rust 1.56 via eselect rust
3. emerge exa
Actual Results:  
Some zombie rustc don't exit and cargo hang

Expected Results:  
cargo won't hang with FEATURES=-sandbox

(Rust 1.56 is not release yet, it will release on 2021-10-21)

Upstream report: https://github.com/rust-lang/rust/issues/89522

> emerge --info
Portage 3.0.26 (python 3.9.7-final-0, default/linux/amd64/17.0/musl, gcc-13.0.0, musl-1.2.2-r3, 5.14.8-zen+ x86_64)
=================================================================
System uname: Linux-5.14.8-zen+-x86_64-Intel-R-_Core-TM-_i9-9900T_CPU_@_2.10GHz-with-libc
KiB Mem:    32645680 total,  11028428 free
KiB Swap:    8388604 total,   7202812 free
Timestamp of repository gentoo: Tue, 05 Oct 2021 04:36:37 +0000
Head commit of repository gentoo: ea4e312c260f3ecbf0f4557a0fc902bfbeb4de43

Timestamp of repository gentoo-zh: Mon, 04 Oct 2021 01:06:16 +0000
Head commit of repository gentoo-zh: 312107f2205ee3543dad93b5539a7d040595f92c

Timestamp of repository guru: Sun, 03 Oct 2021 07:12:26 +0000
Head commit of repository guru: e67b9cf012d1dfd950adc6c4794cc3b28849923c

Timestamp of repository mv: Sat, 02 Oct 2021 09:51:20 +0000
Head commit of repository mv: 252d7490701964c65533e50c316734ab24db4d98

Timestamp of repository pentoo: Tue, 05 Oct 2021 02:36:15 +0000
Head commit of repository pentoo: 02f0b193029e4edf4a3ec70eea71ec984a3f21cb

Timestamp of repository pg_overlay: Tue, 05 Oct 2021 01:06:22 +0000
Head commit of repository pg_overlay: f00613aa0cb8b461326b7ca68f906b29f1687287

Timestamp of repository musl: Sat, 02 Oct 2021 18:21:16 +0000
Head commit of repository musl: 63526da3e64b1b9cb234ce874a471209cd877546

sh bash 5.1_p8
ld LLD 13.0.0 (compatible with GNU linkers)
ccache version 4.4.2 [enabled]
app-shells/bash:          5.1_p8::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.34.0-r2::gentoo
dev-lang/python:          2.7.18_p13::pg_overlay, 3.9.7::pg_overlay
dev-lang/rust:            1.56_beta20210918::12101111-overlay
dev-util/ccache:          4.4.2::gentoo
dev-util/cmake:           3.21.3::gentoo
sys-apps/baselayout:      2.8::gentoo
sys-apps/openrc:          0.44.5::gentoo
sys-apps/sandbox:         2.25::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.71-r1::gentoo
sys-devel/automake:       1.16.5::gentoo
sys-devel/binutils:       2.37_p1::12101111-overlay
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.14::gentoo (virtual/os-headers)
sys-libs/musl:            1.2.2-r3::gentoo
Repositories:

gentoo
    location: /usr/portage/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo
    priority: -1000

gentoo-zh
    location: /usr/portage/gentoo-zh
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo-zh.git
    masters: gentoo

guru
    location: /usr/portage/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo

mv
    location: /usr/portage/mv
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/mv.git
    masters: gentoo

pentoo
    location: /usr/portage/pentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/pentoo.git
    masters: gentoo

pg_overlay
    location: /usr/portage/pg_overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/pg_overlay.git
    masters: gentoo

musl
    location: /usr/portage/musl
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/musl.git
    masters: gentoo
    priority: 900

12101111-overlay
    location: /usr/portage/local
    masters: gentoo
    priority: 1000

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-gentoo-linux-musl"
CC="clang"
CFLAGS="-march=skylake -mtune=skylake -O3 -pipe -falign-functions=32 -fomit-frame-pointer -ffunction-sections -fdata-sections -flto=thin"
CHOST="x86_64-gentoo-linux-musl"
CONFIG_PROTECT="/etc /usr/lib/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXX="clang++"
CXXFLAGS="-march=skylake -mtune=skylake -O3 -pipe -falign-functions=32 -fomit-frame-pointer -ffunction-sections -fdata-sections -flto=thin -stdlib=libc++"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance candy ccache config-protect-if-modified distlocks downgrade-backup ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-march=skylake -mtune=skylake -O3 -pipe -falign-functions=32 -fomit-frame-pointer -ffunction-sections -fdata-sections"
GENTOO_MIRRORS="https://mirrors.tuna.tsinghua.edu.cn/gentoo"
INSTALL_MASK=""
LANG="zh_CN.UTF-8"
LC_ALL="zh_CN.UTF-8"
LDFLAGS="-fuse-ld=lld -rtlib=compiler-rt -unwindlib=libunwind -flto=thin -Wl,--lto-O3 -Wl,-O3 -Wl,--gc-sections -march=skylake -mtune=skylake"
MAKEOPTS="-j10"
PKGDIR="/var/cache/binpkgs"
PORTAGE_BINHOST="ssh://root@192.168.1.104/mnt/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-Ctarget-cpu=skylake -Clinker=clang -Clink-arg=-fuse-ld=lld -Ctarget-feature=-crt-static"
USE="X acl alsa amd64 bluetooth bzip2 clang cli compiler-rt crypt cups dbus default-compiler-rt default-libcxx dri elogind eselect-ldso fortran gif iconv ipv6 jemalloc jpeg libcxx libcxxabi libglvnd libtirpc libunwind llvm-libunwind ncurses nls nptl opengl openmp pam pcre png policykit pulseaudio readline seccomp split-usr ssl svg tcpd unicode vaapi vulkan wayland xattr zlib zsh-completion" ABI_X86="64" ADA_TARGET="gnat_2019" ALSA_CARDS="hda-intel usb-audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="musl" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev libinput" KERNEL="linux" L10N="zh-CN" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="X86 ARM AArch64 RISCV WebAssembly AMDGPU AVR BPF Hexagon Lanai MSP430 Mips NVPTX PowerPC Sparc SystemZ XCore" LUA_SINGLE_TARGET="lua5-4" LUA_TARGETS="lua5-4 luajit" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" QEMU_SOFTMMU_TARGETS="arm aarch64 i386 x86_64 riscv32 riscv64" QEMU_USER_TARGETS="arm aarch64 riscv32 riscv64" RUBY_TARGETS="ruby30" USERLAND="GNU" VIDEO_CARDS="intel i965 iris" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Josh Triplett 2021-10-15 19:15:01 UTC
As far as I can tell, this happens because the sandbox tool has a global lock, and hooks fork to acquire that lock before forking and drop it afterwards (so that fork doesn't happen while another thread holds the lock), but does not have a similar hook for clone or clone3. It's possible to create another process using clone or clone3 (not just a thread), if the flags do not include CLONE_VM. I think the right fix is to hook clone and clone3, and if the flags do *not* include CLONE_VM, use the same lock/unlock logic.
Comment 2 Josh Triplett 2021-10-15 19:54:43 UTC
(Note that since clone3 is a syscall and not a library call, library interposing wouldn't help here. It may be necessary to do external ptracing on programs that use clone3.)
Comment 3 SpanKY gentoo-dev 2021-10-17 00:16:25 UTC
(In reply to Josh Triplett from comment #1)

yes, sandbox relies on the program not making syscalls directly and forking its own processes.  if it does that, all bets are off.  considering how painful it has been to use the clone syscall, no one has bothered, especially when portable code already has access to everything it needs via the fork() & pthread C library APIs.

(In reply to Josh Triplett from comment #2)

we currently only ptrace static programs.  as painful as interposing is, it's still less flaky than ptrace, especially on some architectures.

while detecting if a particular ELF is static is trivial, detecting if it uses a specific syscall is practically impossible.

it'd prob help to have someone dig a bit into rust/cargo source to see where the clone3 call is coming from.  if it were glibc, our existing symbol interposition should have handled it.
Comment 4 SpanKY gentoo-dev 2021-10-17 01:47:26 UTC
there is a clone C library wrapper we could/should interpose and check the flags, but i suspect that isn't the source of the trouble here.  the clone function still hits the clone syscall only, not clone3.  the only way glibc itself hits clone3 is via __clone_internal, and that's only used with pthread & spawn APIs.

so if rust itself is hitting clone3 syscall directly via its internal code, we're a bit buggered.  we could maybe add an env var hook to sandbox to force it to use its ptrace mode, and rust ebuilds/eclasses could set that.
Comment 5 Tiernan Hubble 2021-10-18 03:14:33 UTC
rust 1.56 is indeed doing a direct clone3 syscall: https://github.com/rust-lang/rust/pull/81825/files#diff-7015a38ee6056bbfa832b33281ffeaad5531c4dbfaff60ddfce0934475e040f4R163

Also, while I'm not too familiar with Rust, this change appears to be in the rust standard library (or whatever they call it) - so it would affect anything built with rust, not just cargo. I've already run into a related issue trying to do profile-guided optimization in firefox (which runs firefox in a virtual X server during build, to generate profile data - it hangs forever, just like cargo).

I did manage to "fix" cargo by patching this line to default "HAS_CLONE3" to false rather than true: https://github.com/rust-lang/rust/blob/5e02151318ddd431aea6d58e23948246c1446044/library/std/src/sys/unix/process/process_unix.rs#L147
Comment 6 Tiernan Hubble 2021-10-18 03:27:00 UTC
Looks like they disabled this feature (in most cases) in 1.56 final, so hopefully it won't be an issue: https://github.com/rust-lang/rust/pull/89924

There's still a lot of discussion in the original bug though ( https://github.com/rust-lang/rust/issues/89522 ), so it's not clear what's going to happen in future versions.
Comment 7 SpanKY gentoo-dev 2021-10-18 08:32:21 UTC
we'll have to rework the sandbox startup logic so we only run in ptrace mode.  i'm not sure how much cutting this will take though.

first we'll have to fix a long standing bug where ptrace only works one-level deep.  if the traced process forks, we don't keep track of any of those descendants.  i have this somewhat working.
Comment 8 Josh Stone 2021-10-18 19:36:05 UTC
FWIW, Rust's "direct" clone3 syscall is actually using libc's syscall (2), so in theory you could interpose that and decode number==SYS_clone3. I don't know if it's worth the effort though, because I think Rust is going to have to stop using clone3 anyway, for all the libc arguments against it.
Comment 9 SpanKY gentoo-dev 2021-10-19 04:44:42 UTC
(In reply to Josh Stone from comment #8)

thanks ... i had been including syscall() here, but you're certainly right we could interpose that interface too.  a cursory scan of my system shows that not too many programs go through that, so it shouldn't (hopefully) be too much overhead if we want to go that route.
Comment 10 Georgy Yakovlev archtester gentoo-dev 2021-10-21 19:07:32 UTC
looks like workaround made in in 1.56.0
Comment 11 SpanKY gentoo-dev 2021-10-24 05:19:03 UTC
i have seccomp+ptrace working locally, and i'll prob ship it in sandbox-2.28.  waiting to hear back on the new 2.27 release which includes NNP enabled (and is a requirement for seccomp usage).

however, ptrace is still only used when running static or set*id programs, it isn't used all the time.  will have to do more work to pull the ptrace logic out into the main code path as an option (vs using LD_PRELOAD at all).

we also have the problem that the ptrace support hasn't been ported to all architectures.  i did it for a bunch that i had easy access to via Gentoo dev machines, but that's not all of them.  so i think the LD_PRELOAD logic will be with us for quite some time.

also need to see what kind of perf hit this takes overall.  or maybe it's a win.
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-10-24 05:20:28 UTC
(In reply to SpanKY from comment #11)
> i have seccomp+ptrace working locally, and i'll prob ship it in
> sandbox-2.28.  waiting to hear back on the new 2.27 release which includes
> NNP enabled (and is a requirement for seccomp usage).
> 

yeah, it's been fine so far, but giving it a bit longer is a good idea.

> we also have the problem that the ptrace support hasn't been ported to all
> architectures.  i did it for a bunch that i had easy access to via Gentoo
> dev machines, but that's not all of them.  so i think the LD_PRELOAD logic
> will be with us for quite some time.
> 

which are you missing?
Comment 13 SpanKY gentoo-dev 2021-10-24 11:23:29 UTC
sandbox arch list:
https://gitweb.gentoo.org/proj/sandbox.git/tree/libsandbox/trace/linux?h=v2.27
Gentoo arch list:
https://gitweb.gentoo.org/repo/gentoo.git/tree/profiles/arch.list

at this point, i'm considering implementing generic layers in the kernel and sending those upstream.  linux-5.3 has PTRACE_GET_SYSCALL_INFO which generalizes the reading side of the equation, but there's no generic "set" operation for forcing an error.  then i wouldn't have to fight with sandbox always being behind the curve with newer Linux ports.
Comment 14 SpanKY gentoo-dev 2021-10-26 03:03:41 UTC
updated port status:
* i got an arm64 laptop got that mostly working (i think i hit a kernel bug)
* i got access to the sparc dev box and fixed the sparc32 & sparc64 ports
* i fixed the ppc64 ptrace logic

the big ones we're still missing are mips and maybe riscv.  also m68k & sh, but we know those are lesser used.

talking with some folks about getting a common set ptrace API in the kernel.  optimistic that it'll happen, but it'll be a while before we can rely on it.
Comment 15 SpanKY gentoo-dev 2021-10-26 04:10:01 UTC
relevant ptrace discussion in case people are interested:
https://lists.strace.io/pipermail/strace-devel/2021-October/010745.html

but at this point, i don't think it's a blocker for progress with most arches.
Comment 16 SpanKY gentoo-dev 2021-11-03 00:44:53 UTC
sandbox-3.0 is out w/support for tracing of children of static programs.  that's going to need a while to bake i think.
Comment 17 Georgy Yakovlev archtester gentoo-dev 2021-11-19 20:12:22 UTC
rust thing ported to beta branches, so it's not only in 1.56* now.
https://github.com/rust-lang/rust/pull/90938
Comment 18 Andreas K. Hüttel archtester gentoo-dev 2022-02-12 17:41:11 UTC
so what's the status here regarding glibc-2.34?
Comment 19 SpanKY gentoo-dev 2022-02-17 05:00:35 UTC
pretty sure they reverted all clone3 related changes, so it shouldn't be a blocker anymore.  but i don't know what rust releases, if any, it's part of.
Comment 20 Georgy Yakovlev archtester gentoo-dev 2022-02-17 05:10:46 UTC
rust is ok, all affected releases are gone. one currently in the tree is not affected.
Comment 21 Andreas K. Hüttel archtester gentoo-dev 2022-02-27 21:12:02 UTC
(In reply to SpanKY from comment #19)
> pretty sure they reverted all clone3 related changes, so it shouldn't be a
> blocker anymore.  but i don't know what rust releases, if any, it's part of.

(In reply to Georgy Yakovlev from comment #20)
> rust is ok, all affected releases are gone. one currently in the tree is not
> affected.

OK unblocking then, thanks