While experimenting a bit with OpenMP offloading on GPU, some required wrappers are built but never deployed. Thus: $ usr/lib/llvm/15/bin/clang++ --rocm-device-lib-path=/usr/lib/amdgcn/bitcode --libomptarget-amdgpu-bc-path="/usr/lib64" -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -x c openmp_offload.c clang-15: error: unable to execute command: Executable "clang-linker-wrapper" doesn't exist! clang-15: error: linker command failed with exit code 1 (use -v to see invocation) Building sys-devel/clang step-by-step using ebuild commands shows that the required wrappers are compiled however the install/merge phase ignores them: # find /var/tmp/portage/sys-devel/clang-15.0.7-r1 -iname 'clang-*-wrapper' -type f /var/tmp/portage/sys-devel/clang-15.0.7-r1/work/x/y/clang-abi_x86_64.amd64/bin/clang-nvlink-wrapper /var/tmp/portage/sys-devel/clang-15.0.7-r1/work/x/y/clang-abi_x86_64.amd64/bin/clang-offload-wrapper /var/tmp/portage/sys-devel/clang-15.0.7-r1/work/x/y/clang-abi_x86_64.amd64/bin/clang-linker-wrapper Anyway, manually copying the missing wrapper files to /usr/lib/llvm/15/bin makes the compiler a bit more happy but triggers a crash at the liking step (this is another story). I certainly did something wrong here but the wrappers should be deployed :/
Oh my, FWICS the logic in check_distribution_components() unconditionally ignores all tools. I need to fix that, and then we're going to get a report of what tools we're missing.
Oh wait, I've read that `case` wrong. Either way, I'm going to investigate.
Ok, so the problem is that these packages are "executables" and not "tools", using CMake terminology, so they don't have install targets in distribution build mode. This is an upstream bug. However, FWICS these tools are gone in 17.x, so it's unlikely to be fixed there. HOw important are these? Is it worth patching old releases for that?
Well... So far LLVM production versions are 15 and 16 as LLVM 17 is not released (yet). Is the fix worth it? In theory, yes. How many people uses a Gentoo userland to run programs that make use of OpenMP with offloading to a GPU? No idea :) My *personal* experience with some HPC grids tell me this is not a common scenario. Even finding a good working example is not easy. But in practice, GPU offloading should work as it is shipped with official Gentoo ebuilds. The fix seems trivial from my point of view, just copy some additional files in the staging area and let portage do the rest. Shade: My current knowledge of LLVM being somewhat limited I am perhaps a bit too simplistic here. According to my current comprehension, Gentoo ROCm 5.4 packages make use of LLVM 15 as they define LLVM_MAX_SLOT=15 as of today. In theory, while ROCm 5.3 is "tied" to LLVM 15, ROCm 5.4 supports LLVM 16. Perhaps both works with LLVM 17 (currently masked in Gentoo), to be tested for a potential "happy path". Unfortunately AMD uses a fork of LLVM with their own patches and Gentoo changed the game a bit there to use the system LLVM. Another alternative would be to make ROCm stuff deploy the official AMD fork in a distinct location. More hassle and work but it is possible. Both approaches have their pros and cons. A discussion on the subject here => https://github.com/RadeonOpenCompute/ROCm/issues/1904 Thank you for taking from your the time to investigate it, if I can help in any manner I will be glad to contribute. :)
An OpenMP+GPU offload example can be found here => https://docs.amd.com/bundle/AMD-Instinct-MI100-High-Performance-Computing-and-Tuning-Guide-v5.2/page/Running_Applications_on_AMD_GPUs.html The above example is in attached file.
Created attachment 859874 [details] Usecase that makes use of OpenMP+GPU offload
I haven't come across this bug because I've been developing in Gentoo prefix on a debian host, and debian happens to provide clang-linker-wrapper [1], while not using LLVM_DISTRIBUTION_COMPONENTS. > HOw important are these? Is it worth patching old releases for that? It breaks offloading openmp codes to ROCm/CUDA [2]. I think it's OK to backport https://github.com/llvm/llvm-project/commit/c2aabcfc8395ba30884dd3c1ecc2b192ae85a78d. Also by looking at the generated "${WORKDIR}/x/y/clang-abi_x86_64.amd64/tools/cmake_install.cmake" I see clang-linker-wrapper listed in. I'm curious why it is not installed eventually? [1] https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/blob/16/debian/clang-tools-X.Y.install.in?ref_type=heads#L34 [2] https://wiki.gentoo.org/index.php?title=ROCm&oldid=1217283#OpenMP
Well, FWICS Debian has clang-linker-wrapper only. Are the other two needed as well? I'm mostly asking because apparently they've been removed from main since.
As per description in the source code, I would tend to include the other tools as well: //===-- clang-linker-wrapper/ClangLinkerWrapper.cpp - wrapper over linker-===// [REDACTED] //===---------------------------------------------------------------------===// // // This tool works as a wrapper over a linking job. This tool is used to create // linked device images for offloading. It scans the linker's input for embedded // device offloading data stored in sections `.llvm.offloading` and extracts it // as a temporary file. The extracted device files will then be passed to a // device linking job to create a final device image. // //===---------------------------------------------------------------------===// //===-- clang-offload-wrapper/ClangOffloadWrapper.cpp -----------*- C++ -*-===// [REDACTED] //===----------------------------------------------------------------------===// /// /// \file /// Implementation of the offload wrapper tool. It takes offload target binaries /// as input and creates wrapper bitcode file containing target binaries /// packaged as data. Wrapper bitcode also includes initialization code which /// registers target binaries in offloading runtime at program startup. /// //===----------------------------------------------------------------------===// //===-- clang-offload-bundler/ClangOffloadBundler.cpp ---------------------===// [REDACTED] //===----------------------------------------------------------------------===// /// /// \file /// This file implements a clang-offload-bundler that bundles different /// files that relate with the same source code but different targets into a /// single one. Also the implements the opposite functionality, i.e. unbundle /// files previous created by this tool. /// //===----------------------------------------------------------------------===// //===-- clang-nvlink-wrapper/ClangNvlinkWrapper.cpp - wrapper over nvlink-===// [REDACTED] //===---------------------------------------------------------------------===// /// /// \file /// This tool works as a wrapper over nvlink program. It transparently passes /// every input option and objects to nvlink except archive files. It reads /// each input archive file to extract archived cubin files as temporary files. /// These temp (*.cubin) files are passed to nvlink, because nvlink does not /// support linking of archive files implicitly. /// /// During linking of heterogeneous device archive libraries, the /// clang-offload-bundler creates a device specific archive of cubin files. /// Such an archive is then passed to this tool to extract cubin files before /// passing to nvlink. /// /// Example: /// clang-nvlink-wrapper -o a.out-openmp-nvptx64 /tmp/libTest-nvptx-sm_50.a /// /// 1. Extract (libTest-nvptx-sm_50.a) => /tmp/a.cubin /tmp/b.cubin /// 2. nvlink -o a.out-openmp-nvptx64 /tmp/a.cubin /tmp/b.cubin //===---------------------------------------------------------------------===//
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bf78047aff79cc4eb8cf1661b5fe3472cc21fc04 commit bf78047aff79cc4eb8cf1661b5fe3472cc21fc04 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2023-04-11 18:24:37 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2023-04-11 18:25:01 +0000 sys-devel/clang: Install clang-*-wrapper tools in 15.0.7-r1 Add missing `clang-*-wrapper` tools to 15.0.7-r1. Since these tools aren't crucial to clang operation, let's add them without revbump to avoid forcing a huge rebuild on everyone. The `clang-linker-wrapper` is already installed by 17.x, and I have requested backporting a fix to get it installed into 16.x release branch. The two remaining tools were already removed in 16.x. Bug: https://bugs.gentoo.org/904143 Signed-off-by: Michał Górny <mgorny@gentoo.org> sys-devel/clang/clang-15.0.7-r1.ebuild | 7 +++++++ 1 file changed, 7 insertions(+)
Seems to fix the issue for LLVM 15, good job. Waiting for the upstream PR concerning LLVM 16 to be integrated.
FYI: The upstream pull-request for LLVM 16 is here => https://github.com/llvm/llvm-project-release-prs/pull/415 As of today, it passed all checks and is waiting to be merged.
