Summary: | sys-devel/clang does not install clang-tools-extra | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Paul Preney <paul> |
Component: | Current packages | Assignee: | LLVM support project <llvm> |
Status: | RESOLVED INVALID | ||
Severity: | normal | CC: | matthew, mgorny, paul, sam |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | AMD64 | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: |
Singularity definition file that builds a Gentoo container + some tools.
BASH script that automates building a Gentoo Linux and test case containers. |
Description
Paul Preney
2021-10-16 04:45:39 UTC
https://archives.gentoo.org/gentoo-dev/message/6d3cf88d858fcf3fbb11818ce5d6ea42 So build system of Chapel should try to use -lclang-cpp (or less likely -lclang) instead of those many -lclang* options (-lclangFrontend, -lclangSerialization, ...). I'm sorry but your analysis is wrong. The ebuild does install clang-tools-extra -- e.g. clangd is definitely installed. The ebuild does not install static libraries by design. All public API should be included in libclang-cpp.so. Chapel needs to be fixed to support linking to the single shared library (also called the 'dylib' in upstream LLVM terms). If you hit any problems after replacing the list of linked libraries with '-lclang-cpp', feel free to reopen the bug. Changes needed probably in this file: https://github.com/chapel-lang/chapel/blob/main/third-party/llvm/Makefile.include-system Thanks for the heads-up on libraries to link to: I fixed the Chapel source to link to -lclang-cpp. Such still failed so I also ensured third-party/llvm/Makefile.include-system had the following: LLVM_CLANG_LIBS=-lclang-cpp LLVM_CXXFLAGS=$(LLVM_CONFIG_CXXFLAGS) $(LLVM_MY_CXXFLAGS) -DHAVE_LLVM -fPIC LLVM_CFLAGS=$(LLVM_CONFIG_CFLAGS) -DHAVE_LLVM -fPIC LLVM_LIBS=-dynamic -shared -L$(LLVM_CONFIG_LIB_DIR) $(LLVM_CLANG_LIBS) $(LLVM_LLVM_LIBS) So by starting a new shell and running: wget https://github.com/chapel-lang/chapel/releases/download/1.25.0/chapel-1.25.0.tar.gz tar xvf chapel-1.25.0.tar.gz cd chapel-1.25.0 # update third-party/llvm/Makefile.include-system per above export CHPL_LLVM=system export CHPL_LLVM_CONFIG=/usr/lib/llvm/11/bin/llvm-config export CHPL_HOST_COMPILER=llvm export CHPL_TARGET_COMPILER=llvm source util/setchplenv.bash make -j it successfully compiles but when the chpl program is run (even with no options), it immediately segfaults. (Using other options export CHPL_LIB_PIC=pic option in the above doesn't help either. The -fPIC is needed and shared library options are needed in the LLVM Makefile for the code to compile and link. If either is omitted, it will fail to link.) My current hypothesis is the segfault may be the result of the Chapel code using include files from an older Clang v11 (i.e., what is bundled) than what is installed in the system. I will be investigating this. Created attachment 745935 [details]
Singularity definition file that builds a Gentoo container + some tools.
If you don't already have Singularity installed, "emerge sys-cluster/singularity". See the comment with the demo.sh attachment for additional details.
Created attachment 745938 [details] BASH script that automates building a Gentoo Linux and test case containers. I made a discovery: if sys-devel/clang is built using default options (i.e., the USE flags default-compiler-rt, default-libcxx, and default-lld are not set) then Chapel v1.25.0 builds and works. In various other cases it does not. I made this discovery because I was asked in my Chapel ticket concerning this issue how to install Gentoo Linux so such could be re-created, etc. I thought it would be best to use a container for such and Singularity (i.e., sys-cluster/singularity for those running Gentoo Linux, or, https://singularity.hpcng.org/ https://github.com/hpcng/singularity (or commercial: https://sylabs.io )), thus... I have attached two files: * attachment demo.sh * attachment singularity-gentoo-amd64-chapel-basis.def that completely automate the following: 1. Building a pristine Gentoo Linux (stage3) Singularity container image + some needed packages. This is placed in a directory called gentoo.dir/ by demo.sh. 2. Copying gentoo.dir/ to caseNUM.dir/ and then installing sys-devel/clang:11 + Chapel in the caseNUM.dir/ Singularity container image. Item 1 and each case in item 2 are logged, i.e., gentoo.log and caseNUM.log files. Since each of these are in their own separate directories and are containers one can explore after demo.sh finishes any container by running: * singularity shell -C -w caseNUM.dir (per demo.sh the Chapel stuff was built and installed in /testing/chapel and the rest of the image is Gentoo Linux). Adding more cases (whatever with USE flags, etc.) to demo.sh is straight-forward and will sanely facilitate determining which issues are Chapel issues and/or Gentoo Linux sys-devel/clang-related issues. (I do set MAKEOPTS="-j$(nproc) -l$(nproc)" in the Gentoo Linux build to speed up builds. If such is undesired, it is easily found and edited in both files.) Running demo.sh on a 16-core AMD Ryzen Threadripper PRO 3955WX in RAM takes approximately 30 minutes or so for each case after building the base Gentoo Linux container used by all cases. Each case's log file is about 20M and each case directory is about 3.3G. (gentoo.dir is about 1.9G and gentoo.log about 6.8M.) Chapel is built and successfully runs in case01 and case04. In case01, sys-devel/clang is built using the default USE flags. In case04, sys-devel/clang is built with the default-lld USE flag set. The remaining cases fail as follows: * case02: sys-devel/clang was successfully built with the default-compiler-rt USE flag. Unfortunately, Chapel failed to link at the end with "/usr/bin/x86_64-pc-linux-gnu-ld: ./../build/compiler/linux64/llvm/x86_64/llvm-system/none/AST/AggregateType.o: undefined reference to symbol '_Unwind_Resume@@GCC_3.0' /usr/bin/x86_64-pc-linux-gnu-ld: /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/libgcc_s.so.1: error adding symbols: DSO missing from command line" * case03: sys-devel/clang was successfully built with the default-libcxx USE flag. Unfortunately, Chapel failed to link with the errors that originally triggered me opening this ticket. * case05: sys-devel/clang was successfully built with the default-compiler-rt and default-lld USE flags. Unfortunately, Chapel failed to link with a (different) _Unwind_Resume (undefined symbol) error than case01. * case06: sys-devel/clang was NOT successfully built with the default-compiler-rt, default-libcxx, and default-lld USE flags set due to a block, i.e., "[blocks B ] sys-libs/libunwind ("sys-libs/libunwind" is blocking sys-libs/llvm-libunwind-12.0.1)". This is essentially a stage3 Gentoo Linux install + a couple of tools + attempting to install clang. NOTE: For case06 and this bug report... This sys-devel/clang block is desired behaviour, fine but if not, then something needs to be fixed. (Obviously building Chapel was not done in case06.) NOTE: The above cases write to /etc/portage/package.use/llvm to specify the USE flags for sys-devel/clang. The demo.sh script emerges sys-devel/clang:11 --and somehow that pulls in sys-devel/clang:12. Again, if this is not desired, then perhaps it is something that should be fixed? For completeness (although it is easily seen in demo.sh) building Chapel was done in each case using the following commands: source /etc/profile cd /testing/chapel/ export CHPL_LLVM=system export CHPL_LLVM_CONFIG=/usr/lib/llvm/11/bin/llvm-config export CHPL_HOST_COMPILER=llvm export CHPL_HOST_CC=clang-11 export CHPL_HOST_CXX=clang++-11 export CHPL_TARGET_COMPILER=llvm export CHPL_TARGET_CC=clang-11 export CHPL_TARGET_CXX=clang++-11 sed -i -e 's/LLVM_CLANG_LIBS=.*/LLVM_CLANG_LIBS=-lclang-cpp/' third-party/llvm/Makefile.include-system source util/setchplenv.bash make -j\$(nproc) chpl --version chpl --help The reason I used CHPL_HOST_* and CHPL_TARGET_* was to try to ensure Chapel used Clang version 11 since more than one version of Clang was pulled in through the emerging of sys-devel/clang:11 to try to keep things clean. Finally, in the demo.sh script one should start at the bottom, e.g., build_singularity_base_gentoo_linux_container |& tee -a "gentoo.log" build_working_chapel ${CHAPEL_VERSION} case01 build_working_chapel ${CHAPEL_VERSION} case02 default-compiler-rt build_working_chapel ${CHAPEL_VERSION} case03 default-libcxx build_working_chapel ${CHAPEL_VERSION} case04 default-lld build_working_chapel ${CHAPEL_VERSION} case05 default-compiler-rt default-lld build_working_chapel ${CHAPEL_VERSION} case06 default-compiler-rt default-libcxx default-lld If one wants to re-create a case, simply "sudo rm -rf caseNUM.dir" and then re-run the script (or comment out those lines). Obviously, to run demo.sh one has to have app-admin/sudo installed and configured + sys-cluster/singularity installed first. Many continued thanks for all help, comments, suggestions, etc. :-) Paul P.S. It took a while to create all of this and with the script + the .def file and I hope it is useful for this bug (and perhaps maybe to help explore other bugs/tickets/issues). P.P.S. I opened a Github issue with Chapel concerning building it with a system-installed LLVM/Clang. It is here: https://github.com/chapel-lang/chapel/issues/18589 . I will be appropriately posting there about the demo.sh script, etc. very soon as well. |