[18/36] x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_simple.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_simple.p/scx_simple.c.o -MF scheds/c/scx_simple.p/scx_simple.c.o.d -o scheds/c/scx_simple.p/scx_simple.c.o -c ../scx-1.0.5/scheds/c/scx_simple.c FAILED: scheds/c/scx_simple.p/scx_simple.c.o x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_simple.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_simple.p/scx_simple.c.o -MF scheds/c/scx_simple.p/scx_simple.c.o.d -o scheds/c/scx_simple.p/scx_simple.c.o -c ../scx-1.0.5/scheds/c/scx_simple.c In file included from ../scx-1.0.5/scheds/c/scx_simple.c:13: scheds/c/scx_simple.p/scx_simple.bpf.skel.h: In function ‘scx_simple__create_skeleton’: scheds/c/scx_simple.p/scx_simple.bpf.skel.h:251:12: error: ‘struct bpf_map_skeleton’ has no member named ‘link’ 251 | map->link = &obj->links.simple_ops; | ^~ [19/36] x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_userland.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_userland.p/scx_userland.c.o -MF scheds/c/scx_userland.p/scx_userland.c.o.d -o scheds/c/scx_userland.p/scx_userland.c.o -c ../scx-1.0.5/scheds/c/scx_userland.c FAILED: scheds/c/scx_userland.p/scx_userland.c.o x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_userland.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_userland.p/scx_userland.c.o -MF scheds/c/scx_userland.p/scx_userland.c.o.d -o scheds/c/scx_userland.p/scx_userland.c.o -c ../scx-1.0.5/scheds/c/scx_userland.c In file included from ../scx-1.0.5/scheds/c/scx_userland.c:32: scheds/c/scx_userland.p/scx_userland.bpf.skel.h: In function ‘scx_userland__create_skeleton’: scheds/c/scx_userland.p/scx_userland.bpf.skel.h:268:12: error: ‘struct bpf_map_skeleton’ has no member named ‘link’ 268 | map->link = &obj->links.userland_ops; | ^~ [20/36] /usr/lib/llvm/18/bin/clang -g -O2 -Wall -Wno-compare-distinct-pointer-types -D__TARGET_ARCH_x86 -mcpu=v3 -mlittle-endian '-idirafter /usr/lib/llvm/18/bin/../../../../lib/clang/18/include' '-idirafter /usr/include' -target bpf -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include/vmlinux -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include/bpf-compat -c ../scx-1.0.5/scheds/c/scx_qmap.bpf.c -o scheds/c/scx_qmap.p/scx_qmap.bpf.o [21/36] x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_central.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_central.p/scx_central.c.o -MF scheds/c/scx_central.p/scx_central.c.o.d -o scheds/c/scx_central.p/scx_central.c.o -c ../scx-1.0.5/scheds/c/scx_central.c FAILED: scheds/c/scx_central.p/scx_central.c.o x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_central.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_central.p/scx_central.c.o -MF scheds/c/scx_central.p/scx_central.c.o.d -o scheds/c/scx_central.p/scx_central.c.o -c ../scx-1.0.5/scheds/c/scx_central.c In file included from ../scx-1.0.5/scheds/c/scx_central.c:16: scheds/c/scx_central.p/scx_central.bpf.skel.h: In function ‘scx_central__create_skeleton’: scheds/c/scx_central.p/scx_central.bpf.skel.h:289:12: error: ‘struct bpf_map_skeleton’ has no member named ‘link’ 289 | map->link = &obj->links.central_ops; | ^~ [22/36] /usr/lib/llvm/18/bin/clang -g -O2 -Wall -Wno-compare-distinct-pointer-types -D__TARGET_ARCH_x86 -mcpu=v3 -mlittle-endian '-idirafter /usr/lib/llvm/18/bin/../../../../lib/clang/18/include' '-idirafter /usr/include' -target bpf -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include/vmlinux -I /var/tmp/portage/sys-kernel/scx-1.0.5/work/scx-1.0.5/scheds/include/bpf-compat -c ../scx-1.0.5/scheds/c/scx_flatcg.bpf.c -o scheds/c/scx_flatcg.p/scx_flatcg.bpf.o [23/36] x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_pair.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_pair.p/scx_pair.c.o -MF scheds/c/scx_pair.p/scx_pair.c.o.d -o scheds/c/scx_pair.p/scx_pair.c.o -c ../scx-1.0.5/scheds/c/scx_pair.c FAILED: scheds/c/scx_pair.p/scx_pair.c.o x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_pair.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_pair.p/scx_pair.c.o -MF scheds/c/scx_pair.p/scx_pair.c.o.d -o scheds/c/scx_pair.p/scx_pair.c.o -c ../scx-1.0.5/scheds/c/scx_pair.c In file included from ../scx-1.0.5/scheds/c/scx_pair.c:15: scheds/c/scx_pair.p/scx_pair.bpf.skel.h: In function ‘scx_pair__create_skeleton’: scheds/c/scx_pair.p/scx_pair.bpf.skel.h:306:12: error: ‘struct bpf_map_skeleton’ has no member named ‘link’ 306 | map->link = &obj->links.pair_ops; | ^~ [24/36] x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_nest.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_nest.p/scx_nest.c.o -MF scheds/c/scx_nest.p/scx_nest.c.o.d -o scheds/c/scx_nest.p/scx_nest.c.o -c ../scx-1.0.5/scheds/c/scx_nest.c FAILED: scheds/c/scx_nest.p/scx_nest.c.o x86_64-pc-linux-gnu-gcc -Ischeds/c/scx_nest.p -Ischeds/c -I../scx-1.0.5/scheds/c -I../scx-1.0.5/scheds/include -I/usr/include/bpf/uapi -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -pipe -march=native -pthread -MD -MQ scheds/c/scx_nest.p/scx_nest.c.o -MF scheds/c/scx_nest.p/scx_nest.c.o.d -o scheds/c/scx_nest.p/scx_nest.c.o -c ../scx-1.0.5/scheds/c/scx_nest.c In file included from ../scx-1.0.5/scheds/c/scx_nest.c:15: scheds/c/scx_nest.p/scx_nest.bpf.skel.h: In function ‘scx_nest__create_skeleton’: scheds/c/scx_nest.p/scx_nest.bpf.skel.h:288:12: error: ‘struct bpf_map_skeleton’ has no member named ‘link’ 288 | map->link = &obj->links.nest_ops; | ^~
This makes very little sense and has to do with libbpf, not bpftool. The commit: https://github.com/libbpf/libbpf/commit/be998aa3d41e1f5f83e3e69a71746b785e0a7b8b#diff-ecec02d2345e3666818b41e879a7e9242acbd0575f20f796b2d8b7fe0cf2841b introduced the link member in struct bpf_map_skeleton and is not even in a released version of libbpf yet. This is very likely someone building thir crate against libbpf master.
(In reply to Holger Hoffstätte from comment #1) > This is very likely someone building thir crate against libbpf master. ..or against the libbpof in the kernel, since the link field is also present in e.g. 6.11.3, see: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/tools/lib/bpf/libbpf.h
Fundamentally the problem here is that the build is trying to use a header which is neither in a released upstream version of libbpf, nor our version (which is also out of date). I don't really know how to fix this short of changing our libbpf ebuild to be kernel-based, with a version scheme similar to bpftool - which is terrible, thanks to upstream not understanding version numbers.
Also the link field is clearly part of the next libbpf release only, since the version in the kernel is already at 1.5: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/lib/bpf/libbpf_version.h
Maybe as temporary measure soneone can whip up a libbpf-1.5.0_<date> snapshot from github master and scx can then depend on that.
But I find if I add a constraint, can fix it. So, I open a PR: https://github.com/gentoo/gentoo/pull/38924
I just: - added llvm-19 to scx's LLVM_COMPAT (I no longer have 18) - keyworded libbpf-9999 & installed it - 'ebuild scx-1.0.5.ebuild clean compile' completed successfully. Also the ebuild tells us that bpftool is not even involved: ---snip--- sched_ext schedulers 1.0.5 User defined options ... bpftool : disabled ... ---snip---
Created attachment 905137 [details, diff] Patch for a temporary libbpf-1.5.0_prerelease ebuild This generates a 1.5.0_pre ebuild from 1.4.5.
Created attachment 905147 [details] scx-1.0.5-build/scheds/c/scx_simple.p/scx_simple.bpf.skel.h I think there is more to it. scx-1.0.5 builds fine for me with libbpf 1.4.5 on one system, but I run into the same issue on another system. Attached is scx-1.0.5-build/scheds/c/scx_simple.p/scx_simple.bpf.skel.h from the system where scx-1.0.5 compiles just fine. Here, scx_simple.bpf.skel.h:251 looks different then the one with the compiler error: 250 s->progs[0].name = "simple_select_cpu"; 251 s->progs[0].prog = &obj->progs.simple_select_cpu; 252 s->progs[0].link = &obj->links.simple_select_cpu; That is not saying that we should not go with holger's proposed solution. Just saying that there is maybe to be another relevant component that we are currently missing.
Created attachment 905148 [details] build.log (successful, scx 1.0.5 with libbpf 1.4.5) This the is the build.log from the system where scx 1.0.5 builds successfully with libbpf 1.4.5. Notice the Run-time dependency libbpf found: YES 1.4.5 in the log.
I'll add bug 941120 to See Also in the sense that it's another mysterious BPF failure.
So I don't really know what the scx build does in detail, but I have a vague idea why it may have worked after downgrading bpftool - though it probably should not have. All this is because bpftool is versioned inconsistently, and people use the bpftool output to "do things" like inferring versions & capabilities. The main reason why nobody should be using the old bpftool-6.x anymore is that they produce Gentoo-specific, but incorrect version output. I fixed that _specifically_ because vimproved stumbled over that in https://github.com/gentoo/gentoo/pull/37572. So much for the backstory. It's important to understand that bpftool bundles libbpf (because that's how the kernel-based bpftool build works). bpftool-7.5.0-r2 is based on kernel 6.11.2, which already contains the link member and claims it's at version 1.5.0. bpftool-7.5.0-r1 is based on 6.10.3 and also claims to be 1.5.0, but does NOT contain the link member in bpf_map_skeleton. :-( *If* the scx build uses bpftool (despite the fact that is says it does not, but who knows what the rust parts do..) it might be that it generates something according to its ABI and assumes the bpftool-libbpf is consistent with the unbundled external libbpf it uses for headers. I don't know if that's what's going on, but it could explain that you get different results on different machines and presumably using different bpftool versions? That being said, for scx you really should be using -r2, not -r1, and update the external libbpf to a version that matches the one in bpftool. That's why it built for me with -9999 or the 1.5.0_pre version. It's unfortunate that this is inconsistent. One possible way to reduce this might be to make bpftool use an external libbpf. The out-of-tree versions always trail the in-kernel version by a mile, so we might be forced to use git snapshots occasionally (like the 1.5.0_pre ebuild). I don't really know whether it's actually possible to build bpftool against an external libbpf.
(In reply to Florian Schmaus from comment #10) > Created attachment 905148 [details] > build.log (successful, scx 1.0.5 with libbpf 1.4.5) > > This the is the build.log from the system where scx 1.0.5 builds > successfully with libbpf 1.4.5. Notice the > > Run-time dependency libbpf found: YES 1.4.5 > > in the log. Like I suspected the mseon build also finds and uses bpftool: ... Program bpftool found: YES (/usr/bin/bpftool) ... Which bpftool version is on this machine where it builds with libbpf-1.4.5?
(In reply to Holger Hoffstätte from comment #7) > ... > bpftool : disabled > ... description: 'bpftool to use when generating .bpf.skel.h. By default, bpftool is automatically downloaded and built during setup. To use an existing bpftool binary, point this to the bpftool path or set it to "disabled" to have meson try to find it automatically') So according to newspeak "disabled" is enabled but external.
(In reply to Holger Hoffstätte from comment #13) > Like I suspected the mseon build also finds and uses bpftool: > > ... > Program bpftool found: YES (/usr/bin/bpftool) > ... Yes, it seems that scx requires bpftool at compile time (maybe to create the skel.h header files). Which would also mean that the ebuild is missing a BDEPEND on bpftool. The bpftool=disabled instructs scx's build system to not build bpftool itself, but instead use the system provided one. > Which bpftool version is on this machine where it builds with libbpf-1.4.5? 7.5.0-r1
(In reply to Holger Hoffstätte from comment #13) > Which bpftool version is on this machine where it builds with libbpf-1.4.5? I'm assuming it's 7.5.0-r1 (based on 6.10.x), because I can reproduce it. With an external libbpf-1.4.5 configured by 7.5.0-*r2* scx_simple fails to build due to missing link member. When using *r1* it succeeds because the libbpf in bpftool is based on 6.10.x, does not know about the link member and therefore does not generate it either. Both have "libbpf 1.5" - what even are version numbers? :-( All that being said, just because it builds with -r1 does not mean the scx ebuild should be constrained since it does not really fix the underlying problem (which will just come up again in the future).
Fundamentally this is caused by the disconnect between libbpf-in-bpftool and external libbpf (and its headers). I'm increasingly thinking they should be two ebuilds (to accommodate the different versions) but based on the exact same source: a specific kernel version. I'll poke around in the kernel-based libbpf tree to see how to build it as shared lib.
(In reply to Holger Hoffstätte from comment #17) > I'll poke around in the kernel-based libbpf tree to see how to build it as > shared lib. Well that turned out to be a nothingburger. :) In <kernel source>/tools/lib/bpf just call: make libbpf.so to make the shared lib make libbpf.a to make the static lib make libbpf.pc to make the pkgconfig or just make install prefix=/usr (or "${EPREFIX}"/usr to be Gentoo-ish). ..and you're done. Might need to delete unwanted stuff like static lib depending on USE but that's cherry on top. However I'm not sure how top best couple bpftool and libbpf together since they do not require each other. Suggestions welcome.
Maybe bpftool should just be part of libbpf? We could start by extending the bpftool ebuild, which now already does all the work of unpacking/patching the kernel tree correctly, to also install the lib/headers. We could even call the combined result bpf-tools, in line with xdp-tools, but that's optional. bpftool is pulled in practically everywhere in tandem with libbpf anyway (or transitively e.g. via xdp-tools!), and we can let ebuilds depend on a single package instead of constantly missing the bpftool BDEPEND. One downside would be to find a proper ebuild version number for the new libbpf ebuild, since whatever upstream uses is clearly not meaningful in any way. I really dislike that this coupling is necessary, but the only other alternative is to have libbpf and bpftool with exact same versions and somehow depend on each other, which seems worse.
Regardless of what we do with bpftool vs. libbpf in the future (which should probably be a new bug) I've just created a PR for 1.5.0_pre which should at least fix the build failure with bpftool-7.5.0-r2. It *should* also be backwards-compatible with -r1 since the skeleton code generated by it does not know anything about the link member.
(In reply to Holger Hoffstätte from comment #19) > I really dislike that this coupling is necessary, but the only other > alternative is to have libbpf and bpftool with exact same versions and > somehow depend on each other, which seems worse. Maybe a bit gross but this could be enforced with a virtual/bpf-tools which depends on and enforces a consistent version between the two, and encourage other packages to depend on that at least at build time unless they Re quite sure they only need one of the two.
Not sure if it should be discussed here, in #941248, or if we a further meta tracker bug for the bpf tooling situation. (In reply to Eli Schwartz from comment #21) > Maybe a bit gross but this could be enforced with a virtual/bpf-tools Yes, it's a bit gross but besides co-packaging bpftool & libbpf the only alternative, right? And, if I understand Holger in #941248 correctly, co-packaging bpftool & libbpf also has downsides. However, scx requires AFAIKT bpftool as BDEPEND and libbpf as (R)DEPEND. Using virtual/bpf-tools would mean that we pull in bpftool as RDEPEND, even though it is not required. Maybe this could be improved a bit via a USE flag for virtual/bpf-tools. So, e.g., scx would end up with BDEPEND=virtual/bpf-tools[bpftool] DEPEND=virtual/bpf-tools RDEPEND=virtual/bpf-tools I am not sure if it is worth it, though. Note that I believe this could still fail in a cross-build scenario, because we can not express that the version of bpf-tools in BDEPEND must match the DEPEND version. Hopefully there is a better option. In any case, this is certainly no blocker for me. Just something I wanted to point out. And I am also fine with the co-packaging approach.
(In reply to Florian Schmaus from comment #22) > However, scx requires AFAIKT bpftool as BDEPEND and libbpf as (R)DEPEND. > Using virtual/bpf-tools would mean that we pull in bpftool as RDEPEND, even > though it is not required. Maybe this could be improved a bit via a USE flag > for virtual/bpf-tools. So, e.g., scx would end up with > > BDEPEND=virtual/bpf-tools[bpftool] > DEPEND=virtual/bpf-tools > RDEPEND=virtual/bpf-tools > > I am not sure if it is worth it, though. > > Note that I believe this could still fail in a cross-build scenario, because > we can not express that the version of bpf-tools in BDEPEND must match the > DEPEND version. Just have virtual/bpf-tools in BDEPEND and dev-libs/libbpf in DEPEND/RDEPEND. There is no point having a virtual that does nothing because a USE flag made it only depend on a single fixed package without even || ( ... ) The attempt at a USE flag won't even help. In regular builds, the virtual in BDEPEND and libbpf directly in DEPEND will coalesce to a single libbpf anyway. (And I don't see a useful distinction between that and the virtual in DEPEND and bpftool directly in BDEPEND.) In cross builds, as you pointed out you cannot have BDEPEND="virtual/bpf-tools[bpftool]" DEPEND="virtual/bpf-tools" and assume that the CHOST and CBUILD versions match. Since it doesn't actually help cross builds, forcing the use of a virtual in DEPEND / RDEPEND has no purpose... > Hopefully there is a better option. In any case, this is certainly no > blocker for me. Just something I wanted to point out. And I am also fine > with the co-packaging approach. Running bpftool from DEPEND in src_compile via qemu. More broadly, don't do cross builds, just cross compile in qemu. It's always an option ;) even if actual native cross-builds are much nicer. If there is no other option, there is no other option... Possibly, this can be checked in src_compile via best_version -b and best_version -d. If they match, proceed as normal. If they don't, try qemu in desperation or simply die. The whole thing sounds like enough hassle that you'll want an eclass though...
In order to expdite this in the least complicated way I just pmasked the -r2 version. We can revisit this when upstream has released new bpftool/libbpf versions. I've been working with upstream on getting the bpftool build from Github to actually work correctly, because _of course_ it's inconsistent compared to the kernel-tree build.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fe3d75c072d779ac92c59c4f36cc29f92e22da82 commit fe3d75c072d779ac92c59c4f36cc29f92e22da82 Author: Holger Hoffstätte <holger@applied-asynchrony.com> AuthorDate: 2024-10-11 17:32:11 +0000 Commit: Florian Schmaus <flow@gentoo.org> CommitDate: 2024-10-11 17:59:30 +0000 profiles: mask dev-util/bpftool-7.5.0-r2 Generates ABI-breaking code and would need a matching but yet unreleased libbpf version to build against. Closes: https://bugs.gentoo.org/941185 Signed-off-by: Holger Hoffstätte <holger@applied-asynchrony.com> Closes: https://github.com/gentoo/gentoo/pull/38944 Signed-off-by: Florian Schmaus <flow@gentoo.org> profiles/package.mask | 5 +++++ 1 file changed, 5 insertions(+)
