It seems that systemd uses a fixed $PATH when spawning distcc. As a result, our clang location from /etc/env.d/*llvm isn't respected and distccd is unable to find clang. distcc-3.3 might be unaffected due to its new-ish shadowdir handling.
I'm affected by this. distccmon-gui showed each job submitted to other workers waiting on "receiving" until a timeout. Is there an official workaround? My own kludge, which works for me, is to add EnvironmentFile=/etc/env.d/10llvm-9992 to /etc/systemd/system/distccd.service.d/00gentoo.conf
(In reply to Bernd Feige from comment #1) > I'm affected by this. distccmon-gui showed each job submitted to other > workers waiting on "receiving" until a timeout. > > Is there an official workaround? > > My own kludge, which works for me, is to add > > EnvironmentFile=/etc/env.d/10llvm-9992 > > to /etc/systemd/system/distccd.service.d/00gentoo.conf Then gcc distributed compilation stops working. It works with EnvironmentFile=/etc/profile.env
Created attachment 591276 [details] Set PATH to ROOTPATH: /etc/systemd/system-environment-generators/50_distcc_path Found a better fix now: Place an environment generator file in /etc/systemd/system-environment-generators which sets the PATH to ROOTPATH taken from /etc/profile.env. Without this, I confirmed the PATH in the environment of systemd units (including distccd) to be /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin With it, it becomes the current ROOTPATH: /usr/lib/llvm/9/bin:/usr/lib/llvm/8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin With this, clang works with distccd.
Created attachment 591278 [details] Set PATH to ROOTPATH: /etc/systemd/system-environment-generators/50_distcc_path Sorry, removed misleading comment in the generator
Hmm but this shouldn't be necessary with modern versions of distccd, as they require separate shadow directory for distccd-available compilers and we symlink clang there.
(In reply to Michał Górny from comment #5) > Hmm but this shouldn't be necessary with modern versions of distccd, as they > require separate shadow directory for distccd-available compilers and we > symlink clang there. Well, sys-devel/distcc-3.3.3 here with "eselect compiler-shadow update distccd" and all. I still got "COMPILE_ERROR exit:110" (i.e. Compiler not found) errors after updating clang because the static EnvironmentFile solution I had before did not reflect the change (setting EnvironmentFile=/etc/profile.env does not work because of the "export" key words and is also an overkill). Advantage of using an environment generator is that it auto-updates. Of course it would be nice if the symlinks maintained by eselect compiler-shadow would be sufficient. I doubt that that would be feasible though because the primary whitelisted binary (clang) needs to call other binaries (llvm) which it expects to be in the PATH as well. Shame on me that I didn't look it up in the source right now, but I think that distcc just matches the name with the symlink but still relies on PATH to actually start it?
I have now checked the distcc sources confirming that /usr/lib/distcc is only used to check whether the compiler=command name (which must not contain a slash) is an accessible file in that directory (dcc_check_compiler_whitelist() in serve.c). The symlinks are never actually read. If the check is successful, the command is just executed using execvp() (in exec.c). Thus, the directory in which the binary resides must be in the PATH.
Oh my, that's really silly indeed. Thanks for checking it. Any reason your patch is using ROOTPATH instead of plain PATH? Both should have LLVM paths in them. Also, please use "sed -n -e '/.../p'" instead of the other way around ;-).
(In reply to Michał Górny from comment #8) > Oh my, that's really silly indeed. Thanks for checking it. > > Any reason your patch is using ROOTPATH instead of plain PATH? Both should > have LLVM paths in them. Sure they do. However, the altered PATH is used for all commands started by systemd, not just for distccd, and it would be a security risk to set PATH to the common user PATH. It also is the least invasive change having the desired effect, since the resulting PATH is practically what the overly restricted default for systemd uses, with just llvm and /opt/bin added. I'm not sure how systemd derives that default PATH, maybe we should file a bug requesting to use ROOTPATH... > Also, please use "sed -n -e '/.../p'" instead of the other way around ;-). Thanks, my sed skills are a bit rusty... Upgraded script comes in a second...
Created attachment 591964 [details] Set PATH to ROOTPATH: /etc/systemd/system-environment-generators/50_distcc_path
Created attachment 591966 [details] Set PATH to ROOTPATH: /etc/systemd/system-environment-generators/50_distcc_path Sorry, one more misleading comment fixed...
I think you are associating the wrong idea with ROOTPATH. Originally, it existed to add /sbin & /usr/sbin only for root. However, that's been really painful to our power users, so eventually PATH and ROOTPATH became the same. That said, the only reason LLVM is present in ROOTPATH is that Portage historically used ROOTPATH instead of PATH.
(In reply to Michał Górny from comment #12) > I think you are associating the wrong idea with ROOTPATH. > > Originally, it existed to add /sbin & /usr/sbin only for root. However, > that's been really painful to our power users, so eventually PATH and > ROOTPATH became the same. > > That said, the only reason LLVM is present in ROOTPATH is that Portage > historically used ROOTPATH instead of PATH. Actually ROOTPATH was previously used to construct the default PATH for root in /etc/profile, so it's not my "wrong idea" but, if anything, an idea previously heeded in gentoo. And as far as I can see, it's still maintained separately from PATH, not mainly to avoid 'sbin' in the common user PATH but to avoid paths like /usr/games/bin in root's PATH. It still makes perfect sense to me to keep the PATH for root (and processes running with root privileges) as short as possible. In systemd, a compile-time "search-binaries-default" setting is used (cf. man systemd.service). Whether PATH or ROOTPATH, it seems wrong to me that the system maintains a PATH setting making the installed software work, and then systemd starts services with an arbitrary fixed (and sometimes wrong) value for it. Strangely I couldn't find any bug report on systemd and PATH settings for services...
@systemd, how would you feel about using system-environment-generators to put our regular PATH for all systemd units?
That sounds like a reasonable thing to add to sys-apps/gentoo-systemd-integration. I don't understand why we are using sed to transform profile.env; it's a shell script, so you could just source it and echo the PATH variable. As well, profile.env is POSIX sh compatible, so there's no need to hard-code /bin/bash. Example: #!/bin/sh . /etc/profile.env echo "PATH=$PATH"
(In reply to Mike Gilbert from comment #15) > That sounds like a reasonable thing to add to > sys-apps/gentoo-systemd-integration. > > I don't understand why we are using sed to transform profile.env; it's a > shell script, so you could just source it and echo the PATH variable. As > well, profile.env is POSIX sh compatible, so there's no need to hard-code > /bin/bash. > > Example: > > #!/bin/sh > . /etc/profile.env > echo "PATH=$PATH" Precisely my idea. Do you want to do it, or should I?
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/gentoo-systemd-integration.git/commit/?id=96e0294f0892d355cf6f632b9848c06a2cd2578d commit 96e0294f0892d355cf6f632b9848c06a2cd2578d Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-12-06 19:21:53 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-12-06 19:21:53 +0000 Add env gen to inject full Gentoo PATH to services Bug: https://bugs.gentoo.org/651904 Signed-off-by: Michał Górny <mgorny@gentoo.org> Makefile.am | 5 +++++ system-environment-generators/10-gentoo-path | 3 +++ 2 files changed, 8 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f4cad940e98c5160cf0bb98825dfbd1205199260 commit f4cad940e98c5160cf0bb98825dfbd1205199260 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-12-06 19:31:43 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-12-06 19:36:40 +0000 sys-apps/gentoo-systemd-integration: Bump to v8 Closes: https://bugs.gentoo.org/651904 Signed-off-by: Michał Górny <mgorny@gentoo.org> sys-apps/gentoo-systemd-integration/Manifest | 1 + .../gentoo-systemd-integration-8.ebuild | 32 ++++++++++++++++++++++ 2 files changed, 33 insertions(+)
