Summary: | sys-apps/sandbox-3.0: handing off children to new tracer fails w/yama.ptrace_scope=1: ptrace(PTRACE_ATTACH): Operation not permitted | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Sam James <sam> |
Component: | Sandbox | Assignee: | Sandbox Maintainers <sandbox> |
Status: | CONFIRMED --- | ||
Severity: | normal | CC: | matthew, mozilla, pascal.jaeger |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: | https://bugs.gentoo.org/show_bug.cgi?id=821532 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | 821499, 879087 | ||
Bug Blocks: | |||
Attachments: |
build.log (killed after it hanged)
.config (5.10.76) `cargo --version; rustc +stable` with sandbox debugging on ps faux strace -f sandbox file -L /usr/bin/rust (with debug, killed) strace -f -o log rustc +stable (within sandbox) strace -f -o log sandbox rustc strace -f -o log rustc (no sandbox) rustc |
Description
Sam James
![]() ![]() ![]() ![]() Created attachment 748131 [details]
build.log (killed after it hanged)
I didn't get this error/hang wiht sandbox-2.29. building firefox is working for me w/sandbox-3.0. so can't explain this. we seem to have comparable FEATURES & USE settings. but the failure itself looks pretty basic -- it's just running cargo. do you have YAMA LSM stuff enabled in the kernel ? can you reproduce this directly ? sandbox rustc +stable cargo +stable rustc --version --verbose cargo --version --verbose rustc & cargo are dynamically linked on my system too, so not sure why your system is running these programs through the ptrace code path. Created attachment 748140 [details] .config (5.10.76) (In reply to SpanKY from comment #3) > building firefox is working for me w/sandbox-3.0. so can't explain this. > > we seem to have comparable FEATURES & USE settings. but the failure itself > looks pretty basic -- it's just running cargo. do you have YAMA LSM stuff > enabled in the kernel ? No, not intentionally anyway. Attached config but I'm not aware of having done anything weird. Note that everything works with 2.29 and I feel like I would've noticed if YAMA was messing with ptrace in general before now? $ sysctl kernel.yama.ptrace_scope kernel.yama.ptrace_scope = 1 Let me know if I can grab any other info. > > can you reproduce this directly ? > sandbox > rustc +stable > cargo +stable > rustc --version --verbose > cargo --version --verbose > Good idea, and yes, I can! ``` sam@mop ~ $ sandbox ============================= Gentoo path sandbox ============================== Detection of the support files. Verification of the required files. Setting up the required environment variables. The protected environment has been started. -------------------------------------------------------------------------------- Process being started in forked instance. sam@mop ~ $ rustc +stable * /var/tmp/portage/sys-apps/sandbox-3.0/work/sandbox-3.0/libsandbox/trace.c:_do_ptrace():83: failure (Operation not permitted): * ISE:_do_ptrace: ptrace(PTRACE_ATTACH, ..., 0x0000000000000000, 0x0000000000000000): Operation not permitted ``` > rustc & cargo are dynamically linked on my system too, so not sure why your > system is running these programs through the ptrace code path. This is what threw me straight away, I don't get why ptrace is even involved. $ file -L /usr/bin/cargo /usr/bin/cargo: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, stripped No idea if this is interesting or relevant but even if I temporarily set SANDBOX_METHOD (wasn't set before this, just trying it for fun), ptrace is used? sam@mop ~ $ grep SANDBOX_METHOD /etc/sandbox.conf # SANDBOX_METHOD SANDBOX_METHOD="preload" sam@mop ~ $ rustc +stable * /var/tmp/portage/sys-apps/sandbox-3.0/work/sandbox-3.0/libsandbox/trace.c:_do_ptrace():83: failure (Operation not permitted): * ISE:_do_ptrace: ptrace(PTRACE_ATTACH, ..., 0x0000000000000000, 0x0000000000000000): Operation not permitted (In reply to Sam James from comment #5) Again, might be completely useless, but weirdly it works when run inside of gdb. $ gdb --args sandbox rustc +stable [...] Reading symbols from sandbox... Reading symbols from /usr/lib/debug//usr/bin/sandbox.debug... (gdb) r Starting program: /usr/bin/sandbox rustc +stable [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [Detaching after fork from child process 9419] error: couldn't read +stable: No such file or directory (os error 2) error: aborting due to previous error [Inferior 1 (process 9415) exited with code 01] great that we can isolate from firefox try running `export SANDBOX_{VERBOSE,DEBUG}=yes` inside of `sandbox` and before running `cargo --version`. see if that has anything interesting in it. Created attachment 748143 [details]
`cargo --version; rustc +stable` with sandbox debugging on
Attached the full thing although none of it looks that interesting?
(In reply to Sam James from comment #8) > * tracing: /usr/bin/rustc assuming `file -L /usr/bin/rustc` says it's dynamically linked, that shouldn't have happened. can you compress+attach that here ? also can you run (outside of sandbox) `strace -f -o log rustc +stable` and attach that here too. i'm guessing this behavior didn't start with 3.0, it just got worse because the only diff between 2.29 & 3.0 is that we trace children of traced processes. Created attachment 748146 [details] ps faux (In reply to Sam James from comment #8) > Created attachment 748143 [details] > `cargo --version; rustc +stable` with sandbox debugging on > > Attached the full thing although none of it looks that interesting? fwiw, rustc +stable is the only thing from the list which triggers it, which is weird, but think I was slightly misled here, it doesn't hang when calling cargo, but rustc (obvious from the above). portage 36634 3.3 0.2 47048 41832 pts/2 SN 04:04 0:01 | \_ /var/tmp/portage/www-client/firefox-94.0/work/firefox_build/_virtualenvs/build/bin/python /var/tmp/portage/www-client/firefox-94.0/work/firefox-94.0/configure.py portage 36838 0.0 0.2 47048 36972 pts/2 SN 04:04 0:00 | \_ /var/tmp/portage/www-client/firefox-94.0/work/firefox_build/_virtualenvs/build/bin/python /var/tmp/portage/www-client/firefox-94.0/work/firefox-94.0/configure.py portage 36839 0.0 0.1 168968 26264 pts/2 SNl 04:04 0:00 | \_ /usr/bin/rustc +stable portage 36841 0.1 0.0 0 0 pts/2 ZN 04:04 0:00 | \_ [python] <defunct> Created attachment 748149 [details] strace -f sandbox file -L /usr/bin/rust (with debug, killed) (In reply to SpanKY from comment #9) Attach what, sorry? Just stracing it inside of sandbox? That's really dull both ways: ``` $ strace -f -o log sandbox rustc +stable * absolute_path: /proc/self/statm * resolved_path: /proc/39340/statm * ACCESS ALLOWED: open_rd: /proc/self/statm * absolute_path: /opt/rust-bin-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib * resolved_path: /opt/rust-bin-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib * ACCESS ALLOWED: opendir: /opt/rust-bin-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib * EARLY FAIL: open64(+stable): No such file or directory error: couldn't read +stable: Invalid argument (os error 22) error: aborting due to previous error ``` ``` sam@mop ~ $ sandbox strace -f -o log rustc +stable * absolute_path: /proc/sys/kernel/pid_max * resolved_path: /proc/sys/kernel/pid_max * ACCESS ALLOWED: open_rd: /proc/sys/kernel/pid_max * absolute_path: /home/sam/log * resolved_path: /home/sam/log * ACCESS PREDICTED: fopen_wr: /home/sam/log strace: Can't fopen 'log': Permission denied ``` But this is fun: sam@mop ~ $ export SANDBOX_{VERBOSE,DEBUG}=yes sam@mop ~ $ file -L /usr/bin/rustc * absolute_path: /usr/bin/file * resolved_path: /usr/bin/file * ACCESS ALLOWED: execve: /usr/bin/file * absolute_path: /usr/share/misc/magic.mgc Bad system call Only happens when debugging is enabled and probably a distraction. Created attachment 748152 [details]
strace -f -o log rustc +stable (within sandbox)
(In reply to Sam James from comment #11) attach the rustc binary itself (not the symlink) so i can analyze it attach the strace of running rustc *outside* of the sandbox so i can compare it to your log running inside i know about the file SIGSYS crash. it's unrelated, and only happens inside of file. i added a fsync() call to libsandbox which file's seccomp bpf filter is not expecting. it's safe to ignore for this bug. Created attachment 748155 [details]
strace -f -o log sandbox rustc
Created attachment 748158 [details]
strace -f -o log rustc (no sandbox)
Created attachment 748161 [details]
rustc
oh, i see why now. this rustc you have implements its own set of memory hooks that do not play safely with an in-process sandbox. see http://crbug.com/586444 for gory details. did you compile this yourself ? or is it a prebuilt ? i can reproduce the SANDBOX_METHOD=preload not being respected, so let me fix that too while we're here. (In reply to SpanKY from comment #17) > did you compile this yourself ? or is it a prebuilt ? pre-built: dev-lang/rust-bin-1.56.1::gentoo was built with the following: USE="verify-sig -clippy -doc (-prefix) -rls -rustfmt" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="sse2" > > i can reproduce the SANDBOX_METHOD=preload not being respected, so let me > fix that too while we're here. cheers! opened tab for reading of the tcmalloc bug later too. heading off for a bit now but it sounds like you've got enough to work with atm. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=373c81e05db464d82d9f667871d682b36804de15 commit 373c81e05db464d82d9f667871d682b36804de15 Author: Mike Frysinger <vapier@gentoo.org> AuthorDate: 2021-11-03 04:50:10 +0000 Commit: Mike Frysinger <vapier@gentoo.org> CommitDate: 2021-11-03 04:50:10 +0000 sandbox: fix passing of config env vars down This code has been buggy since it was first added years ago -- it would read the right value out of the config file, but then always just set $SANDBOX_VERBOSE to it instead of the right env var. This prevented the basic loading of sandbox settings from sandbox.conf. Bug: https://bugs.gentoo.org/821403 Signed-off-by: Mike Frysinger <vapier@gentoo.org> src/environ.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=7c92fad8b8e613ada5b4ce951829ed420a4aaac7 commit 7c92fad8b8e613ada5b4ce951829ed420a4aaac7 Author: Mike Frysinger <vapier@gentoo.org> AuthorDate: 2021-11-03 04:56:17 +0000 Commit: Mike Frysinger <vapier@gentoo.org> CommitDate: 2021-11-03 04:56:17 +0000 libsbutil: drop fsync when logging This was added as part of running multiple tracers in parallel in the hopes (hack) it would make logs less intermingled. Unfortunately, it didn't really accomplish that, and it upsets `file` when verbose output is enabled due to file's own seccomp filter (which doesn't have fsync). We could add this to file's seccomp filter (since it's a pretty benign syscall), but easier to just drop it at this point since it's not all that useful. Bug: https://bugs.gentoo.org/821403 Signed-off-by: Mike Frysinger <vapier@gentoo.org> libsbutil/sb_efuncs.c | 1 - 1 file changed, 1 deletion(-) ok, we've fixed the side bugs, but we still have the original one here to contend with. why is the ptrace hand off failing with EPERM ? let me try installing the prebuilt to see if i can reproduce that failure. i was afraid installing rust-bin was going to take a while due to conflicts, but i guess these install in parallel fine. that said, it's working for me :/. $ sandbox $ rustc --version rustc 1.56.1 (59eed8a2a 2021-11-01) $ export SANDBOX_{DEBUG,VERBOSE}=yes $ rustc --version * absolute_path: /usr/bin/rustc * resolved_path: /opt/rust-bin-1.56.1/bin/rustc-bin-1.56.1 * ACCESS ALLOWED: execve: /usr/bin/rustc * tracing: /usr/bin/rustc ... i don't have YAMA enabled in my kernel though, so let me try that next. ok, rebuilding my kernel with extra options didn't help. so i went back and just read the code harder. i think i have a handle on where it's going wrong, and i think i can force it to fail in the same way as you. ok, i lied a bit. the issue does seem to be yama related. the default for yama/ptrace_scope is 1. if i set that to 0, i can't make it fail anymore. i can't reproduce with the rust prebuilt, but i can with some custom test case. yama is a common setting i think, so i'll have to figure out how to make this work. in the mean time, i'll have the children tracing auto-disable if yama is greater than 1, and get a sandbox-3.1 out with that mitigation in place. firefox-94 compilation fails for me with funny sandbox violation. ACCESS DENIED: open_wr: /usr/share/fonts/nerd-fonts Do you need any logs/help from me? [ebuild R ] sys-apps/sandbox-3.0::gentoo USE="nnp" ABI_X86="(32) (64) (-x32)" 0 KiB [ebuild R ] dev-lang/rust-1.56.1:stable/1.56::gentoo USE="rustfmt system-bootstrap system-llvm -clippy -debug -doc -miri -nightly (-parallel-compiler) -rls -test -verify-sig -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_TARGETS="AMDGPU (X86) -AArch64 -ARM -AVR -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -WebAssembly -XCore" 0 KiB [ebuild U ] www-client/firefox-94.0:0/94::gentoo [93.0:0/93::gentoo] USE="clang dbus gmp-autoupdate hwaccel lto openh264 pgo pulseaudio screencast system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland -debug -eme-free -geckodriver -hardened -jack (-selinux) -sndio -wifi" L10N="en-GB pl -ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -ca-valencia -cak -cs -cy -da -de -dsb -el -en-CA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -ne -nl -nn -oc -pa -pt-BR -pt-PT -rm -ro -ru -sco -si -sk -sl -son -sq -sr -sv -szl -ta -te -th -tl -tr -trs -uk -ur -uz -vi -xh -zh-CN -zh-TW" 0 KiB The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=746d68ae5972575d5fd87b7bd82e318d56352d9e commit 746d68ae5972575d5fd87b7bd82e318d56352d9e Author: Mike Frysinger <vapier@gentoo.org> AuthorDate: 2021-11-03 06:40:08 +0000 Commit: Mike Frysinger <vapier@gentoo.org> CommitDate: 2021-11-03 06:40:08 +0000 libsandbox: add YAMA checks and skip ptrace when active The YAMA ptrace_scope knob restricts access to different ptrace calls depending on the capabilities the current process holds. For now, do not try to ptrace processes when the YAMA level is incompatible with the capabilities that we have. This means we basically cannot protect against processes when they get into this state, so for now, we release them rather than abort. Bug: https://bugs.gentoo.org/771360 Bug: https://bugs.gentoo.org/821403 Signed-off-by: Mike Frysinger <vapier@gentoo.org> libsandbox/trace.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) (In reply to Marek Bartosiewicz from comment #25) > firefox-94 compilation fails for me with funny sandbox violation. > > ACCESS DENIED: open_wr: /usr/share/fonts/nerd-fonts > > Do you need any logs/help from me? > Please file a new bug for this with the full logs as this is something else. i think current FEATURES set factors in too. if it's all building as root (i.e. full caps), then this wouldn't be a problem. but if we have FEATURES="userpriv usersandbox", i think that's when this situation shows up. it could be made to work by having the tracer inject a syscall into the tracee such that it calls prctl(PR_SET_PTRACER) on the new tracer. then we'd be able to attach. if we update portage to pass ambient caps to us that include ptrace, then we'd be able to run fine too. that would prob be easier to do right now. it wouldn't be an issue if i went with a one-tracer-to-many-tracees design, but based on what i've seen in other projects that used that design, the performance tanks pretty quickly since the single tracer process becomes a huge bottleneck. use of ptrace seccomp would make the scale better, but not go away. i don't know if making it threaded changes the equation or security checks. the docs talk about PIDs, not TIDs, but that might be an oversight. would have to dig into the kernel implementation to find out. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0410a141e0a13133097d422078f5edaf2894dc1a commit 0410a141e0a13133097d422078f5edaf2894dc1a Author: Sam James <sam@gentoo.org> AuthorDate: 2021-11-04 00:31:20 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-11-04 00:32:13 +0000 profiles: mask newer sandbox versions w/ ptrace issues Masking for now to avoid duplicate reports of known issues; hangs when emerging e.g. Firefox and seemingly anything Go based are nasty. One of the bugs (821403) should be OK now but tagging anyway for reference. Bug: https://bugs.gentoo.org/821532 Bug: https://bugs.gentoo.org/821523 Bug: https://bugs.gentoo.org/821403 Signed-off-by: Sam James <sam@gentoo.org> profiles/package.mask | 8 ++++++++ 1 file changed, 8 insertions(+) |