Created attachment 506742 [details] /proc/cpuinfo on my PC Steps to reproduce: * Download UndoDB (a proprietary debugger based on GDB): * Open https://undo.io/products/undodb/ * Scroll to the bottom of page * Fill in email address, press "Get free trial" * Find a tarball download link in email, download (~100 MB) * Untar e.g. to /tmp: "tar -xzpf undodb.tgz -C /tmp" * With glibc older than 2.26 (e.g. 2.25-r9), try to debug /bin/echo to see that it works: 194d4ff71373 / # /tmp/undodb-4.7.5397/udb /bin/echo udb: UndoDB reversible debugging system. Copyright 2006-2017 Undo Ltd. udb: Using pre-built package-gdb: /tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb udb: undodb-4.7.5397 udb: Licensed to: <andrey_utkin@fastmail.com> udb: License URL: http://licensing.undo.io/cgi-bin/cgihandler?p=2ce41f8da43ce244a238cb8a udb: udb: The license agreement which you will need to accept to run UndoDB will be shown shortly. While it is being shown, press SPACE to scroll forward, "q" to quit. Press return now to start showing the license. udb: Do you agree to this license? [y or n] y udb: Starting /tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb GNU gdb (GDB) 7.7 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /bin/echo...(no debugging symbols found)...done. (udb) run udb: debug-server pid 28, port 49529 Starting program: /bin/echo undodb: license type: UndoDB version 4, demo, user: warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? Program received signal SIGSTOP, Stopped (signal). 0x00007fd37a95efd8 in _exit () from /lib64/libc.so.6 udb: The program has exited, but is still being debugged. udb: (You may use undodb commands to go backwards.) (udb) * Install glibc-2.26-r3: ACCEPT_KEYWORDS="~amd64" emerge -1 =sys-libs/glibc-2.26-r3 * Run that debugger application again: $ /tmp/undodb-4.7.5397/udb /bin/echo udb: UndoDB reversible debugging system. Copyright 2006-2017 Undo Ltd. udb: Using pre-built package-gdb: /tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb udb: undodb-4.7.5397 udb: Licensed to: <andrey_utkin@fastmail.com> udb: License URL: http://licensing.undo.io/cgi-bin/cgihandler?p=2ce41f8da43ce244a238cb8a udb: By running this software you agree to the terms in: udb: /tmp/undodb-4.7.5397/demo_license.html udb: Starting /tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb GNU gdb (GDB) 7.7 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /bin/echo...done. (udb) run udb: debug-server pid 21423, port 40193 Starting program: /bin/echo undodb: license type: UndoDB version 4, demo, user: warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? Program received signal SIGILL, Illegal instruction. _dl_runtime_resolve_xsavec () at ../sysdeps/x86_64/dl-trampoline.h:118 118 xsavec STATE_SAVE_OFFSET(%rsp) (udb) bt full #0 _dl_runtime_resolve_xsavec () at ../sysdeps/x86_64/dl-trampoline.h:118 No locals. #1 0x000055b8e0487615 in main (argc=1, argv=0x7ffe980b1148) at src/echo.c:112 display_return = true allow_options = <optimized out> do_v9 = <optimized out> (udb) set disassemble-next-line on (udb) show disassemble-next-line Debugger's willingness to use disassemble-next-line is on. (udb) stepi Program received signal SIGILL, Illegal instruction. _dl_runtime_resolve_xsavec () at ../sysdeps/x86_64/dl-trampoline.h:118 118 xsavec STATE_SAVE_OFFSET(%rsp) => 0x00007fd342f04148 <_dl_runtime_resolve_xsavec+104>: 0f c7 (bad) 0x00007fd342f0414a <_dl_runtime_resolve_xsavec+106>: 64 fs 0x00007fd342f0414b <_dl_runtime_resolve_xsavec+107>: 24 40 and $0x40,%al udb: The program has exited, but is still being debugged. udb: (You may use undodb commands to go backwards.) (udb) This seems dependent on my CPU, because it doesn't reproduce on my cloud VPS. My CPU has "xsavec" in flags, and VPS doesn't. I have installed latest microcode update for my CPU, and it didn't affect the issue. /proc/cpuinfo of both machines is attached to the bugticket. I do not use any fancy CFLAGS. The issue reproduces even with gentoo docker container. I am not sure where the issue lies, is it * my CPU being faulty - I appreciate if anybody tries to reproduce this on their xsavec-enabled CPU * glibc generating invalid instruction (maybe not instruction itself invalid, maybe it's parameter being invalid? no idea) * UndoDB doing something too tricky I can ask UndoDB developers to look at this if we get any reason to think the issue may be on their part.
Created attachment 506744 [details] /proc/cpuinfo on my VPS
Oddly, this doesn't reproduce on ArchLinux in docker, with glibc version "2.26-6": $ docker run -v /:/host -it --cap-add=SYS_PTRACE base/archlinux /bin/bash [root@86af02f152da /]# pacman -Q glibc warning: database file for 'core' does not exist warning: database file for 'extra' does not exist warning: database file for 'community' does not exist glibc 2.26-6 [root@86af02f152da /]# /host/tmp/undodb-4.7.5397/udb /bin/echo udb: UndoDB reversible debugging system. Copyright 2006-2017 Undo Ltd. udb: Using pre-built package-gdb: /host/tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb udb: undodb-4.7.5397 udb: Licensed to: <andrey_utkin@fastmail.com> udb: License URL: http://licensing.undo.io/cgi-bin/cgihandler?p=2ce41f8da43ce244a238cb8a udb: udb: The license agreement which you will need to accept to run UndoDB will be shown shortly. While it is being shown, press SPACE to scroll forward, "q" to quit. Press return now to start showing the license. udb: Do you agree to this license? [y or n] y udb: INFO: For best internal diagnostics please install your distribution's addr2line. udb: Starting /host/tmp/undodb-4.7.5397/package-gdb/install/x64/bin/gdb GNU gdb (GDB) 7.7 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /bin/echo...(no debugging symbols found)...done. (udb) run udb: debug-server pid 27, port 59001 Starting program: /usr/bin/echo undodb: license type: UndoDB version 4, demo, user: warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? Program received signal SIGSTOP, Stopped (signal). 0x00007fe499d6f5b8 in _exit () from /usr/lib/libc.so.6 udb: The program has exited, but is still being debugged. udb: (You may use undodb commands to go backwards.) (udb)
Do you you get any other programs to crash?
(In reply to Sergei Trofimovich from comment #3) > Do you you get any other programs to crash? Literally debugging every binary with "udb" leads to same outcome. Besides udb, I do not experience any issues running glibc-2.26-r3 on my workstation.
(In reply to Andrey Utkin from comment #4) > Besides udb, I do not experience any issues running glibc-2.26-r3 on my > workstation. However, I am not sure any other program I normally run uses "xsavec" instruction. Also not sure how to write correct program which does use that instruction.
_dl_runtime_resolve is used by every dynamic glibc-linked program. glibc provides a bunch of _dl_runtime_resolve implementations depending on what kernel reports as supported on your system (via auxv argument) https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/dl-machine.h;h=905a37a5cc596f504f95c17ec1cc42e9747cce7b;hb=HEAD#l122 Note the 120 if (GLRO(dl_x86_cpu_features).xsave_state_size != 0) 121 *(ElfW(Addr) *) (got + 2) 122 = (HAS_ARCH_FEATURE (XSAVEC_Usable) 123 ? (ElfW(Addr)) &_dl_runtime_resolve_xsavec 124 : (ElfW(Addr)) &_dl_runtime_resolve_xsave); If your programs work just fine both when ran as-is and under vanilla gdb I would suspect udb to do something fancy with auxv (or executable loading).
You can try to check how the xsavec bit is initialized in glibc: $ gdb --args /lib64/ld-linux.so.2 /bin/ls (gdb) break _start Breakpoint 1 at 0xab0 (gdb) run Starting program: /lib64/ld-linux.so.2 /bin/ls Breakpoint 1, 0xf7fd7ab0 in _start () (gdb) break get_common_indeces Breakpoint 2 at 0xf7fee6f0 (gdb) continue And peek at how 171 cpu_features->feature[index_arch_XSAVEC_Usable] 172 |= bit_arch_XSAVEC_Usable; gets initialized. https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86/cpu-features.c;h=87aaa8683ccd300392f75d8c2023180ed82d1511;hb=HEAD#l171
(In reply to Sergei Trofimovich from comment #6) > If your programs work just fine both when ran as-is and under vanilla gdb > I would suspect udb to do something fancy with auxv (or executable loading). Thanks for looking into this. I tend to agree. Will ask udb devs. Just wondering why it doesn't reproduce on ubuntu artful and on latest archlinux. Respective docker images feature 2.26 glibc.
Turned out to be a known issue with UndoDB. Thanks for your support, and my excuses for the noise. BTW I couldn't get cpu_features even on very debuggy CFLAGS, always getting it optimized out: ... Breakpoint 1, get_common_indeces (family=0x7fffffffd064, model=0x7fffffffd068, extended_model=0x7fffffffd06c, stepping=0x7fffffffd078, cpu_features=<optimized out>) at ../sysdeps/x86/cpu-features.c:34 ...
(In reply to Andrey Utkin from comment #9) > Turned out to be a known issue with UndoDB. > Thanks for your support, and my excuses for the noise. > > BTW I couldn't get cpu_features even on very debuggy CFLAGS, always getting > it optimized out: > > ... > Breakpoint 1, get_common_indeces (family=0x7fffffffd064, > model=0x7fffffffd068, extended_model=0x7fffffffd06c, > stepping=0x7fffffffd078, cpu_features=<optimized out>) at > ../sysdeps/x86/cpu-features.c:34 > ... glibc ebuild filters out most of optimisation flags because they usually tend to break early dynamic loader. What I usually do is manually add __attribute__((noinline)) to interesting functions.
