ALLOWED_FLAGS currently lacks the following clang specific flags which belong into the same categories as existing ones: Debug targets: -gdbx -glldb -gsce (what even is that?) unlike for gdb, these do not come in level flavors like -ggdb3 More debug: -gmodules There's more specific -g flags at https://clang.llvm.org/docs/ClangCommandLineReference.html#debug-information-generation , but I'm pretty sure those are subflags in -g[0-3] anyways. Those could be added too, but I left them out in favor of readability. Optimization flags: -Oz Currently -O[12sg] are enabled - -Oz is a clang specific and can be summarized as a more aggressive -Os Additionally, the following seems to be missing (gcc & clang): -fuse-ld=* Right now only -fuse-ld is allowed, but neither gcc nor clang have that option without the = behind it? Lastly, the following is missing (gcc & clang): -ffixed-x18 (This flag also exists with other registers, but x18 is the only one I'm aware where it's actually needed) This causes the compiler to not use the x18 register (arm64) in operations. Hand coded assembly can still use x18, and no breakage is expected since x18 is untouched by the PCS. x18 is defined as a platform ABI register by AAPCS64. It is reserved by many OSes and thus primarily relevant for crossdev and prefix. The following platforms reserve x18: Windows on ARM64: x18 points to the TEB - https://docs.microsoft.com/en-us/cpp/build/arm64-windows-abi-conventions?view=msvc-160#integer-registers MacOS, iOS & other Apple OSes on ARM64: "The platforms reserve register x18" - it is unknown what for exactly - https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms Android on ARM64: x18 is used for SCS - https://developer.android.com/ndk/guides/abis#arm64-v8a Additionally, we would like to bring SCS to Gentoo one day, and would also need x18 for that. Best regards, Jannik
(In reply to Jannik Glückert from comment #0) > ALLOWED_FLAGS currently lacks the following clang specific flags which > belong into the same categories as existing ones: Can you clarify what you would like to achieve? The idea of strip-flags is to get a small subset of CFLAGS that is very likely understood both by current and built compilers. Ebuilds should not normally use strip-flags. If they do we probably already break something by aggressive flag filtering. > Debug targets: > -gdbx > -glldb > -gsce (what even is that?) > unlike for gdb, these do not come in level flavors like -ggdb3 What will happen when you build gcc with CXX=clang++ and CFLAGS=-gdbx? I would expect gcc build to fail when we pass -gdbx to gcc-stage1. We could add USE=custom-cflags to gcc to allow people pass suspicious flags there. But it should not be a default
> Can you clarify what you would like to achieve? Make life easier for users with specific needs without compromising the experience for others & fix deficiencies in this eclass > The idea of strip-flags is to get a small subset of CFLAGS that is very likely understood both by current and built compilers. Is it? It looks like the purpose of strip-flags, as utilized by ebuilds right now, is to remove "ricer" custom flags on sensitive packages, but leave debug and specific workaround flags intact. As it is, if a user wants to utilize clang system wide, strip-flags is preventing them from debugging with lldb in some places. > What will happen when you build gcc with CXX=clang++ and CFLAGS=-gdbx? likely not much, because gcc is full of GNU C and only builds with gcc, regardless of clang-only debug flags. What will happen if I build glibc with any of the -fsanitize flags? It'll likely trigger a sanitizer in some arcane UB code from the 80s. What will happen if I build gcc with "-fuse-ld bfd" (allowed by eclass, wrong syntax) instead of "-fuse-ld=bfd" (disallowed, correct syntax)? > gcc: error: unrecognized command-line option '-fuse-ld' Protecting users from mistakes is definitely a thing we should try to do, but if someone wants to smuggle "broken" flags through strip-flags they can easily do so. Someone setting -g* or -fsanitize* is likely doing so for a specific reason, and not because of ricer fashion or curiosity - and right now the eclass is preventing them from doing so. Similarly, someone using -Oz is likely aware that it is a clang only flag, and probably smart enough to figure out that gcc doesn't build with clang after the first attempt. As for -fuse-ld=* and -ffixed-x18: Those are general defects in the eclass and not compiler specific, so they should be fixed regardless of how gentoo interprets the purpose of strip-flags
(In reply to Jannik Glückert from comment #2) > > Can you clarify what you would like to achieve? > > Make life easier for users with specific needs without compromising the > experience for others & fix deficiencies in this eclass Do you have specific examples where it was a hindrance? > > The idea of strip-flags is to get a small subset of CFLAGS that is very likely understood both by current and built compilers. > > Is it? It looks like the purpose of strip-flags, as utilized by ebuilds > right now, is to remove "ricer" custom flags on sensitive packages, but > leave debug and specific workaround flags intact. At the very least we should improve the documentation for strip-flags, it's goals, typical use cases and pitfalls. > As it is, if a user wants to utilize clang system wide, strip-flags is > preventing them from debugging with lldb in some places. > > > What will happen when you build gcc with CXX=clang++ and CFLAGS=-gdbx? > > likely not much, because gcc is full of GNU C and only builds with gcc, > regardless of clang-only debug flags. gcc can be build by any c++11 compiler. Try: # CC=clang CXX=clang++ emerge -v1 sys-devel/gcc The problem of gcc ebuild is in the fact that ebuild calls both $CXX and inplace-built gcc. Both (actually mostly gcc, not $CXX in conrast to many other ebuilds) use $CXXFLAGS. And it better be working if we want make.conf:CC=clang CXX=clang++ CXXFLAGS="-clang-specific-flags" systems to Just Work. They are expected to Just Work today. > What will happen if I build glibc with any of the -fsanitize flags? It'll > likely trigger a sanitizer in some arcane UB code from the 80s. I invite you to try it yourself in a chroot and try to find the cause of breakage. I can help in debugging it. My bet it will not be an UB in C code. The problem of glibc is not a source-level UB. But the fact that C standard has not enough features to implement a runtime for standard C. For one example glibc's runtime linker better not to call out to any external symbols (like ones from libasan.so) until relocations are fully set up. It's actually a big challenge to achieve even on -O0 without any sanitizers. For another example you really don't want your libc.so.6 to be linked against libasan.so (or even underlinked against it). You will get an unbootable system. Similar goes for gcc's libgcc_s.so & co (like, libasan.so itself). > What will happen if I build gcc with "-fuse-ld bfd" (allowed by eclass, wrong syntax) > > gcc: error: unrecognized command-line option '-fuse-ld' If LDFLAGS='-fuse-ld bfd' can't build any ebuilds I think it's fine not to build the ones that use 'strip-flags' either. 'strip-flags' and friends don't handle multiargument options. That's very good reason to avoid stripping when possible. > instead of "-fuse-ld=bfd" (disallowed, correct syntax)? What makes you think it's disallowed? # LDFLAGS=-fuse-ld=bfd emerge -v1 sys-libs/glibc ... x86_64-pc-linux-gnu-gcc ... -fuse-ld=bfd -nostdlib -nostartfiles -static -o .../work/build-amd64-x86_64-pc-linux-gnu-nptl/elf/ldconfig ... You can play with the filter in eclass/tests/flag-o-matic.sh: --- a/eclass/tests/flag-o-matic.sh +++ b/eclass/tests/flag-o-matic.sh @@ -106,6 +106,12 @@ strip-flags [[ -z ${CFLAGS}${LDFLAGS}${CPPFLAGS} && ${CXXFLAGS} == "-O2" ]] ftend +tbegin "strip-flags preserved -fuse-ld=" +CXXFLAGS+=" -fuse-ld=hello " +strip-flags +[[ -z ${CFLAGS}${LDFLAGS}${CPPFLAGS} && ${CXXFLAGS} == "-O2 -fuse-ld=hello" ]] +ftend + tbegin "replace-flags basic" CFLAGS="-O0 -foo" replace-flags -O0 -O1 gentoo/eclass/tests $ ./flag-o-matic.sh * strip-flags: CXXFLAGS: changed '-O2 -fuse-ld=hello ' to '-O2 -fuse-ld=hello' [ ok ] > Protecting users from mistakes is definitely a thing we should try to do We don't filter out user's mistakes. We usually: - bring C*FLAGS to a state that they could be used for a compiler different from $CXX (gcc, firefox with USE=clang, maybe wine?, a few embedded packages with cross-compilers) - remove flags that are too special to be able to break assumptions of fragile programs (glibc, ghc, firefox, wine, chromium? gtk+?) "too special" will certainly be very vague: it will depend of package version (and bugs it has), on compiler version (and bugs and behaviour it introduces with magic flags). Thus 'strip-flags' will always be a very blunt hammer. > but if someone wants to smuggle "broken" flags through strip-flags they can > easily do so. Someone setting -g* or -fsanitize* is likely doing so for a > specific reason, and not because of ricer fashion or curiosity - and right > now the eclass is preventing them from doing so. Similarly, someone using > -Oz is likely aware that it is a clang only flag, and probably smart enough > to figure out that gcc doesn't build with clang after the first attempt. Does not USE=custom-cflags Just Work for such cases? Maybe concrete examples from the past will help me understand why current system is so limiting for you. We can also consider an escape hatch like EXTRA_ALLOWED_FLAGS="" that user could specify if they really want to pass arbitrary flags with big red warnings in the build log when use of one of them is noticed. But at this point it's better not to filter any flags and just expose USE=custom-cflags. > As for -fuse-ld=* and -ffixed-x18: > Those are general defects in the eclass and not compiler specific, so they > should be fixed regardless of how gentoo interprets the purpose of > strip-flags -ffixed-* might be fine to list. I don't immediately see a use case for it though. -fuse-ld=* should already work I believe.
> Do you have specific examples where it was a hindrance? Personally, when I had to build gcc with -ffixed-x18 to work on SCS implementations. A more realistic usecase would be when someone wants to debug e.g. emacs, libreoffice or libsdl with lldb without having to go full custom-cflags or worse, editing the ebuild if it doesn't provide the option. > gcc can be build by any c++11 compiler. Sorry, I was thinking of gcc itself, not the bootstrap compiler - or is that also clang-able now? > Does not USE=custom-cflags Just Work for such cases? Yes, but custom-cflags in general means "you're off the rails now and doing something you probably shouldn't be doing, things will probably break horribly" - but this doesn't apply to the extra debug targets like -glldb > What makes you think it's disallowed? I guess I didn't understand the check function fully - allowing fuse-ld instead of fuse-ld=* (like we do for e.g. -fsanitize*) seemed unintuitive, but if this works there's no urgent reason to fix it. > -ffixed-* might be fine to list. I don't immediately see a use case for it though. As mentioned, without -ffixed-x18 we violate the ABI of any non linux / BSD aarch64 target - it's needed in ALLOWED_FLAGS especially so it doesn't leak into libgcc. We will also need it in the future because we (sam and me) would like to bring SCS to aarch64 gentoo one day. The other -ffixed-* flags should remain out of ALLOWED_FLAGS, there's no usecase since they're all either scratch or procedure call standard regs that are used on all platforms. > My bet it will not be an UB in C code. This was more of an example of how users can easily slip breaking flags through strip-flags, not glibc specific. > For one example glibc's runtime linker better not to call out > to any external symbols (like ones from libasan.so) > until relocations are fully set up. pre-reloc is it's own circle of hell and I do not wish it upon anyone. Now back to topic: > bring C*FLAGS to a state that they could be used for a compiler different from $CXX Is strip-flags really effective at this? There are sanitizer flags that only gcc implements and vice versa for clang - same with debug flags as this request is about. Nonetheless, a user might require these flags to debug a package, without wanting to go full custom-cflags (which may also have some weird side implications in some ebuilds), or not being able to do so at all without modifying the ebuild. People won't go around putting -glldb or -fsanitize=something_only_gcc_implements in their make.conf and then compile everything with it out of spite - and if they do, they'll probably be able to understand the very clear compiler error that will appear immediately, and not some hours within compilation. I don't quite get the "has to work with all compilers" requirement - from what I understand, ALLOWED_FLAGS are mostly flags that users might need to work around specific issues with their setup or platform, or that are needed to debug specific things? Using clang with lldb instead of gdb would certainly fall under these special setups.
(In reply to Jannik Glückert from comment #4) > > Do you have specific examples where it was a hindrance? > > Personally, when I had to build gcc with -ffixed-x18 to work on SCS > implementations. That sounds reasonable. > A more realistic usecase would be when someone wants to > debug e.g. emacs, libreoffice or libsdl with lldb without having to go full > custom-cflags or worse, editing the ebuild if it doesn't provide the option. Only now I realized #530070 already allows -fsanitize= in flag-o-matic.eclass making all my arguments moot :) > > -ffixed-* might be fine to list. I don't immediately see a use case for it though. > > As mentioned, without -ffixed-x18 we violate the ABI of any non linux / BSD > aarch64 target - it's needed in ALLOWED_FLAGS especially so it doesn't leak > into libgcc. Out of curiosity which CHOST (or -march/-mabi?) is that? Sounds like ideally gcc's build system already needs to set it explicitly if it's a target requirement (as opposed to manual register picking by an end user). > We will also need it in the future because we (sam and me) would like to > bring SCS to aarch64 gentoo one day. > > The other -ffixed-* flags should remain out of ALLOWED_FLAGS, there's no > usecase since they're all either scratch or procedure call standard regs > that are used on all platforms. Picking a random register should not cause runtime problems. Worst case there will be a build time failure to allocate the register. Why not allow that? > > My bet it will not be an UB in C code. > > This was more of an example of how users can easily slip breaking flags > through strip-flags, not glibc specific. Ah, I think I was reading it backwards assuming that -fsanitize= was not allowed. That makes sense now. > Now back to topic: > > > bring C*FLAGS to a state that they could be used for a compiler different from $CXX > > Is strip-flags really effective at this? There are sanitizer flags that only > gcc implements and vice versa for clang - same with debug flags as this > request is about. > Nonetheless, a user might require these flags to debug a package, without > wanting to go full custom-cflags (which may also have some weird side > implications in some ebuilds), or not being able to do so at all without > modifying the ebuild. > > People won't go around putting -glldb or > -fsanitize=something_only_gcc_implements in their make.conf and then compile > everything with it out of spite - and if they do, they'll probably be able > to understand the very clear compiler error that will appear immediately, > and not some hours within compilation. > > > I don't quite get the "has to work with all compilers" requirement - from > what I understand, ALLOWED_FLAGS are mostly flags that users might need to > work around specific issues with their setup or platform, or that are needed > to debug specific things? Using clang with lldb instead of gdb would > certainly fall under these special setups. Yeah. Let's change the description to something like: strip-flags: - attempts to filter out most flags that could cause subtle and hard to debug runtime failures (flag examples: -flto, -O3) - does NOT attempt to keep flags that cause package build breakages due to unsupported flags like -gunsupported.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a5e21edfc41b5663110f0a6dddf38975038c7009 commit a5e21edfc41b5663110f0a6dddf38975038c7009 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-01-18 16:50:23 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-01-22 22:18:54 +0000 flag-o-matic.eclass: allow -ffixed-x18 for arm64 Needed for shadow stack bits on ARM64. Closes: https://bugs.gentoo.org/800533 Thanks-to: Jannik Glückert <jannik.glueckert@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> eclass/flag-o-matic.eclass | 3 +++ 1 file changed, 3 insertions(+) Additionally, it has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=47d20223ebfdd16c71f48fec1a6194d4a5e87096 commit 47d20223ebfdd16c71f48fec1a6194d4a5e87096 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-01-18 16:48:02 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-01-22 22:18:54 +0000 flag-o-matic.eclass: allow -glldb We already allow -ggdb for GDB and this is the analogue for LLDB. Bug: https://bugs.gentoo.org/800533 Reported-by: Jannik Glückert <jannik.glueckert@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> eclass/flag-o-matic.eclass | 1 + 1 file changed, 1 insertion(+)
