I'd like to request adding polly [1]. polly is a part of the llvm projects. It's their 'Framework for High-Level Loop and Data-Locality Optimizations'. As such, it is quite similar to gcc's Graphite. As it is part of the llvm-project [2], I guess it should be buildable using the llvm.org eclass as well. [1] polly.llvm.org [2] https://github.com/llvm/llvm-project/tree/master/polly
*** Bug 771138 has been marked as a duplicate of this bug. ***
As a playground I modified the LLVM ebuild with an use flag to include polly support. For now, due my limited knowledge about the LLVM build system, it is for the moment built statically. Someone could take a look for building it as a library? I tried to modify distribution_components functions to add it, but somehow its libraries are not installed. Anyway this static version works in my tests and can be a good place to start?
Created attachment 697146 [details] LLVM ebuild LLVM ebuild with static polly support via USE flag
Note that attached ebuild will make polly to work in currently installed Clang but will break every LLVM dependant install due to not installing Polly LLVMPolly and PollyISL libraries
(In reply to David Carlos Manuelda from comment #4) > Note that attached ebuild will make polly to work in currently installed > Clang but will break every LLVM dependant install due to not installing > Polly LLVMPolly and PollyISL libraries Could you please describe the steps to reproduce the problem with your ebuild? I ported your modifications to LLVM 12.0.0, but it seems to work both using Clang and rustc. I will use it for a few more days to see if something comes up.
(In reply to Leonardo Neumann from comment #5) > (In reply to David Carlos Manuelda from comment #4) > > Note that attached ebuild will make polly to work in currently installed > > Clang but will break every LLVM dependant install due to not installing > > Polly LLVMPolly and PollyISL libraries > > Could you please describe the steps to reproduce the problem with your > ebuild? > I ported your modifications to LLVM 12.0.0, but it seems to work both using > Clang and rustc. I will use it for a few more days to see if something comes > up. The ebuild I posted will work, however, once installed/merged, other LLVM dependant software will fail at configure stage due to missing files. To reproduce, install LLVM with this changes and then try to emerge clang, at Cmake's configure stage it will complain about some .a files not installed by the ebuild.
Created attachment 701571 [details] Polly 12.0.0 Well, I managed to get polly as a standalone package and it seems to work fine and pass on all the supported tests. However, it has the drawback that you have to load it as an external optimization pass, which I think that can be solved by properly linking llvm against polly. To build a program using clang with standalone polly, you can do something like this: clang -Xclang -load -Xclang libPolly.so -O2 -mllvm -polly test.c
(In reply to Leonardo Neumann from comment #7) > Created attachment 701571 [details] > Polly 12.0.0 > > Well, I managed to get polly as a standalone package and it seems to work > fine and pass on all the supported tests. However, it has the drawback that > you have to load it as an external optimization pass, which I think that can > be solved by properly linking llvm against polly. To build a program using > clang with standalone polly, you can do something like this: > > clang -Xclang -load -Xclang libPolly.so -O2 -mllvm -polly test.c I also saw that ebuild and tested that way. Unfortunatelly using it this way breaks many packages, specially some autotools based. The best would be to build LLVM with support for it but done in the proper way otherwise the build process would be a nightmare: Build first LLVM+clang Build polly Rebuild LLVM (+clang?) to link against it
I've implemented Polly several months ago without ebuilds modifications.
(In reply to David Carlos Manuelda from comment #8) > Unfortunatelly using it this way breaks many packages, specially some > autotools based. It shouldn't break anything, as long as the user can provide custom compiler flags. But I agree that linking against LLVM should be way more practical and we are not too far from achieving that. (In reply to Perfect Gentleman from comment #9) > I've implemented Polly several months ago without ebuilds modifications. I stumbled upon your implementation, but unfortunately I didn't manage to get the tests running. Does it fix the linking problems with clang and other LLVM-dependent programs?
(In reply to Leonardo Neumann from comment #10) > I stumbled upon your implementation, but unfortunately I didn't manage to > get the tests running. Does it fix the linking problems with clang and other > LLVM-dependent programs? Hmm... It's needed to add some extra LDFLAGS for LLVM staff.
(In reply to Perfect Gentleman from comment #9) > I've implemented Polly several months ago without ebuilds modifications. Hi PG, how can I do this? I'd like to be able to test Polly. Thank you.
(In reply to unhappy-ending from comment #12) > (In reply to Perfect Gentleman from comment #9) > > I've implemented Polly several months ago without ebuilds modifications. > > Hi PG, how can I do this? I'd like to be able to test Polly. Thank you. I think you need to rebuild LLVM adding the Polly libraries -lPolly -lPollyISL in linker. It works up to LLVM 12. In LLVM 13 I'm having linking error because lld is picking the 32-bit library instead of the 64-bit in the 64-bit compile phase, throwing an error of being incompatible with elf64-x86-64 (I've adapted the ebuild to be multilib).
Created attachment 757373 [details] Polly 13.0.0 The linker flags are the only missing thing to get Polly to work without the annoying plugin loading flags. I've been experimenting with Perfect Gentleman's ebuild for a while, and did some slight modifications, as both LLVM_POLLY_LINK_INTO_TOOLS and LLVM_CMAKE_PATH are not required for the ebuild to work. Also, the LDFLAGS are only required on the LLVM package, as both clang and rustc forwards polly optimizations to LLVM. Please test this ebuild and report whether it works for you.
The only step that I think it would be important now is to add a polly useflag into LLVM ebuild so it pulls Polly as a dependency and also adds the required LDFLAGS for it to work.
(In reply to Leonardo Neumann from comment #14) > Created attachment 757373 [details] > Polly 13.0.0 Tried your ebuild, built polly and tried to rebuild llvm --- ld.lld: error: unable to find library -lPolly ld.lld: error: unable to find library -lPollyISL ---
(In reply to Perfect Gentleman from comment #16) > (In reply to Leonardo Neumann from comment #14) > > Created attachment 757373 [details] > > Polly 13.0.0 > > Tried your ebuild, built polly and tried to rebuild llvm > --- > ld.lld: error: unable to find library -lPolly > ld.lld: error: unable to find library -lPollyISL > --- Those libraries should be available under /usr/lib/llvm/13/lib/libPolly{ISL}.a, just like in your ebuild. I only removed LLVM_POLLY_LINK_INTO_TOOLS that should add the LDFLAGS in the LLVM tools if you build both LLVM and Polly together. Can you please check the installation image if these files are available?
(In reply to Leonardo Neumann from comment #17) > Those libraries should be available under > /usr/lib/llvm/13/lib/libPolly{ISL}.a, just like in your ebuild. I only > removed LLVM_POLLY_LINK_INTO_TOOLS that should add the LDFLAGS in the LLVM > tools if you build both LLVM and Polly together. Can you please check the > installation image if these files are available? No, my ebuild installs them into /usr/lib64 ------- /usr/lib64/libPolly.a /usr/lib64/libPollyISL.a -------
(In reply to Perfect Gentleman from comment #18) > (In reply to Leonardo Neumann from comment #17) > > Those libraries should be available under > > /usr/lib/llvm/13/lib/libPolly{ISL}.a, just like in your ebuild. I only > > removed LLVM_POLLY_LINK_INTO_TOOLS that should add the LDFLAGS in the LLVM > > tools if you build both LLVM and Polly together. Can you please check the > > installation image if these files are available? > > No, my ebuild installs them into /usr/lib64 > ------- > /usr/lib64/libPolly.a > /usr/lib64/libPollyISL.a > ------- ebuild for polly-12.0 installed there
Created attachment 861218 [details] Building LLVM Polly 15 without ebuild modification. Removed GPGPU codegen and installation of related libraries.
Created attachment 861219 [details] Building LLVM Polly 16 without ebuild modification
Just add the files I've included to /etc/portage/env/sys-devel/ and you'll be able to build Polly as part of LLVM without having to modify the ebulids.
Why not add support for building Polly along with LLVM via a USE-flag, similar to GCC and Graphite?
Comment on attachment 861219 [details] Building LLVM Polly 16 without ebuild modification >MYCMAKEARGS+='-DLLVM_ENABLE_PROJECTS="polly" ' >MYCMAKEARGS+='-DLLVM_DISTRIBUTION_COMPONENTS=$(printf "%s;%s" $(get_distribution_components) "Polly;LLVMCore;LLVMScalarOpts;LLVMInstCombine;LLVMTransformUtils;LLVMAnalysis;LLVMipo;LLVMMC;LLVMPasses;LLVMLinker;LLVMIRReader;LLVMBitReader;LLVMMCParser;LLVMObject;LLVMProfileData;LLVMTarget;LLVMTargetParser;LLVMVectorize;LLVMBinaryFormat;LLVMRemarks;LLVMAsmParser;LLVMBitstreamReader;LLVMAggressiveInstCombine;LLVMBitWriter;LLVMFrontendOpenMP;LLVMInstrumentation;LLVMBinaryFormat;LLVMDebugInfoCodeView;LLVMTextAPI;LLVMAsmPrinter;LLVMSymbolize;LLVMDebugInfoDWARF;LLVMCodeGen;LLVMCoroutines;LLVMIRPrinter;LLVMObjCARCOpts;LLVMDebugInfoMSF;LLVMDebugInfoPDB;LLVMSelectionDAG;LLVMExtensions;") ' >MYCMAKEARGS+='-DLLVM_TOOL_POLLY_BUILD=ON -DLLVM_POLLY_LINK_INTO_TOOLS=ON ' >LLVM_COMPONENTS=( llvm cmake polly ) > >function post_src_install() { > function polly_install() { > DESTDIR=${D} cmake_build tools/polly/install/strip > } > > multilib_foreach_abi polly_install >}
Comment on attachment 861218 [details] Building LLVM Polly 15 without ebuild modification. Removed GPGPU codegen and installation of related libraries. >MYCMAKEARGS+='-DLLVM_ENABLE_PROJECTS="polly" ' >MYCMAKEARGS+='-DLLVM_DISTRIBUTION_COMPONENTS=$(printf "%s;%s" $(get_distribution_components) "Polly;LLVMCore;LLVMScalarOpts;LLVMInstCombine;LLVMTransformUtils;LLVMAnalysis;LLVMipo;LLVMMC;LLVMPasses;LLVMLinker;LLVMIRReader;LLVMBitReader;LLVMMCParser;LLVMObject;LLVMProfileData;LLVMTarget;LLVMVectorize;LLVMBinaryFormat;LLVMRemarks;LLVMAsmParser;LLVMBitstreamReader;LLVMAggressiveInstCombine;LLVMBitWriter;LLVMFrontendOpenMP;LLVMInstrumentation;LLVMBinaryFormat;LLVMDebugInfoCodeView;LLVMTextAPI;LLVMAsmPrinter;LLVMSymbolize;LLVMDebugInfoDWARF;LLVMCodeGen;LLVMCoroutines;LLVMObjCARCOpts;LLVMDebugInfoMSF;LLVMDebugInfoPDB;LLVMSelectionDAG;LLVMExtensions;") ' >MYCMAKEARGS+='-DLLVM_TOOL_POLLY_BUILD=ON -DLLVM_POLLY_LINK_INTO_TOOLS=ON ' >LLVM_COMPONENTS=( llvm cmake polly third-party ) > >function pre_src_unpack() { > LLVM_COMPONENTS=( llvm cmake polly third-party) > llvm.org_src_unpack >} > >function post_src_install() { > function polly_install() { > DESTDIR=${D} cmake_build tools/polly/install/strip > } > > multilib_foreach_abi polly_install >}
Created attachment 863850 [details] llvm modified ebuild LLVM-15 ebuild modified to enable polly and lto as IUSE
(In reply to Seryoga Leshii from comment #25) > Comment on attachment 861218 [details] > Building LLVM Polly 15 without ebuild modification. Removed GPGPU codegen > and installation of related libraries. > > >MYCMAKEARGS+='-DLLVM_ENABLE_PROJECTS="polly" ' > >MYCMAKEARGS+='-DLLVM_DISTRIBUTION_COMPONENTS=$(printf "%s;%s" $(get_distribution_components) "Polly;LLVMCore;LLVMScalarOpts;LLVMInstCombine;LLVMTransformUtils;LLVMAnalysis;LLVMipo;LLVMMC;LLVMPasses;LLVMLinker;LLVMIRReader;LLVMBitReader;LLVMMCParser;LLVMObject;LLVMProfileData;LLVMTarget;LLVMVectorize;LLVMBinaryFormat;LLVMRemarks;LLVMAsmParser;LLVMBitstreamReader;LLVMAggressiveInstCombine;LLVMBitWriter;LLVMFrontendOpenMP;LLVMInstrumentation;LLVMBinaryFormat;LLVMDebugInfoCodeView;LLVMTextAPI;LLVMAsmPrinter;LLVMSymbolize;LLVMDebugInfoDWARF;LLVMCodeGen;LLVMCoroutines;LLVMObjCARCOpts;LLVMDebugInfoMSF;LLVMDebugInfoPDB;LLVMSelectionDAG;LLVMExtensions;") ' > >MYCMAKEARGS+='-DLLVM_TOOL_POLLY_BUILD=ON -DLLVM_POLLY_LINK_INTO_TOOLS=ON ' > >LLVM_COMPONENTS=( llvm cmake polly third-party ) > > > >function pre_src_unpack() { > > LLVM_COMPONENTS=( llvm cmake polly third-party) > > llvm.org_src_unpack > >} > > > >function post_src_install() { > > function polly_install() { > > DESTDIR=${D} cmake_build tools/polly/install/strip > > } > > > > multilib_foreach_abi polly_install > >} I am creating ebuilds based in your work
Created attachment 863851 [details] llvm:15[polly?,lto?] LLVM:15 ebuild with polly and lto available as IUSE (to mimic gcc with graphite and lto IUSE)
Created attachment 863852 [details] llvm:16[polly?,lto?] llvm:16 polly and lto available as IUSE
Now the problem with my ebuilds is that I can not compile clang with polly enabled. Will investigate further
(In reply to David Carlos Manuelda from comment #30) > Now the problem with my ebuilds is that I can not compile clang with polly > enabled. Will investigate further Fixed as of missing LLVMExtensions in export, will fix the ebuilds now. Instead of "maintainer wanted" can this be addressed by the @llvm herd so it is permanent? since only llvm ebuild needs to be changed.
Created attachment 863857 [details] llvm-15.0.7-r3[polly?,lto?]
Created attachment 863858 [details] llvm-16.0.6[polly?,lto?]
Created attachment 864049 [details] llvm-15.0.7-r3[polly?,lto?] Ebuild cleaned and now use distribution_components to be installed instead of a post_src_install
Created attachment 864050 [details] llvm-16.0.6[polly?,lto?] Ebuild cleaned and now use distribution_components to be installed instead of a post_src_install
There are some strip errors which are non fatal but annoying as they only happen in the newly added files for Polly. Could someone look and try to fix that so the ebuild is as correct as possible? I don't have enough knowledge beyond this point.
Created attachment 864070 [details] llvm-15.0.7-r3[polly?,lto?] Correctly install Polly files
Created attachment 864071 [details] llvm-16.0.6[polly?,lto?] Correctly install Polly files
Dear David! Can You please fix ebuild for LLVM version 17.0.6? I've renamed 16.0.6 but got errors while configured: CMake Error: install(EXPORT "LLVMExports" ...) includes target "LLVMCodeGen" which requires target "LLVMCodeGenTypes" that is not in any export set. CMake Error: install(EXPORT "LLVMExports" ...) includes target "LLVMSelectionDAG" which requires target "LLVMCodeGenTypes" that is not in any export set. CMake Error: install(EXPORT "LLVMExports" ...) includes target "LLVMAsmPrinter" which requires target "LLVMCodeGenTypes" that is not in any export set. CMake Error: install(EXPORT "LLVMExports" ...) includes target "LLVMSymbolize" which requires target "LLVMDebugInfoBTF" that is not in any export set. CMake Error: install(EXPORT "LLVMExports" ...) includes target "LLVMNVPTXCodeGen" which requires target "LLVMCodeGenTypes" that is not in any export set. Thanks!
I might be more practical to switch to the version of LLVM from xarblu's overlay which then links to the Polly package from said overlay if the `polly` USE flag is enabled: https://gpo.zugaina.org/Overlays/xarblu-overlay/sys-devel/polly https://gpo.zugaina.org/Overlays/xarblu-overlay/sys-devel/llvm https://github.com/xarblu/xarblu-overlay Note: As far as I can tell, when Polly is built externally, `-Xclang -load -Xclang libPolly.so` is needed, but I've not needed that with the packages listed above for whatever reason.
(In reply to Sandi (Sandy) Vujaković from comment #40) > I might be more practical to switch to the version of LLVM from xarblu's > overlay which then links to the Polly package from said overlay if the > `polly` USE flag is enabled: > https://gpo.zugaina.org/Overlays/xarblu-overlay/sys-devel/polly > https://gpo.zugaina.org/Overlays/xarblu-overlay/sys-devel/llvm > https://github.com/xarblu/xarblu-overlay > > Note: As far as I can tell, when Polly is built externally, `-Xclang -load > -Xclang libPolly.so` is needed, but I've not needed that with the packages > listed above for whatever reason. This happens when polly and LLVM are compiled in separate stage(packages) then you need to indicate clang to load polly library. In the ebuilds I made, polly is built along with LLVM and LLVM is linked against it so there is no need for extra clang indications (which IMHO is a more correct aproach). I will prepare soon a PR with updated ebuilds (starting from 18+) so they are reviewed and accepted by Gentoo team.
Since I am not the original reported I kindly ask for 2 changes: * Change title to: sys-devel/llvm: Add polyhedral optimizations support (polly IUSE) * Add https://github.com/gentoo/gentoo/pull/36119 PR to "see also" section Hope this can be reviewed and accepted by Gentoo devs.
