Many `*cmake` and `*CMakeLists.txt` files from the `caffe2` package are hardcoded paths. If we do not call `hprefixify` on those files, cmake will try to find the libs from the host system instead of the prefix (as far as I know). The same problem occurs on `sci-libs/pytorch`, where `tools/setup_helpers/env.py` needs `hprefixify`. Reproducible: Always Steps to Reproduce: 1.create a Gentoo Prefix 2.emerge sci-libs/pytorch Actual Results: failed to emerging `sci-libs/caffe2` and `sci-libs/pytorch` Expected Results: `sci-libs/pytorch` emerged output of `emerge --info`: Portage 3.0.49 (python 3.11.4-final-0, default/linux/amd64/17.1/no-multilib/prefix/kernel-3.2+, gcc-11, glibc-2.37-r3, 6.1.0-10-amd64 x86_64) ================================================================= System uname: Linux-6.1.0-10-amd64-x86_64-13th_Gen_Intel-R-_Core-TM-_i9-13900K-with-glibc2.37 KiB Mem: 65400140 total, 12246864 free KiB Swap: 1000444 total, 986364 free Timestamp of repository gentoo: Sat, 29 Jul 2023 14:00:01 +0000 Head commit of repository gentoo: ec7630170f1f7c47826d6c0738f705ca516cf237 sh bash 5.1_p16-r6 ld GNU ld (Gentoo 2.40 p5) 2.40.0 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r6::gentoo dev-lang/perl: 5.36.0-r2::gentoo dev-lang/python: 3.11.4::gentoo dev-util/cmake: 3.26.4-r1::gentoo dev-util/meson: 1.1.1::gentoo sys-apps/baselayout: 2.13-r1::gentoo sys-apps/sandbox: 2.37::gentoo sys-devel/autoconf: 2.71-r6::gentoo sys-devel/automake: 1.16.5-r1::gentoo sys-devel/binutils: 2.40-r5::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/gcc: 11.3.1_p20230427::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/make: 4.4.1-r1::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.37-r3::gentoo Repositories: gentoo location: /opt/gentoo/var/db/repos/gentoo sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-rsync-verify-jobs: 1 sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: no sync-rsync-extra-opts: Inanna location: /opt/gentoo/var/db/repos/Inanna masters: gentoo volatile: True localrepo location: /opt/gentoo/var/db/repos/localrepo masters: gentoo volatile: True ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /opt/gentoo/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /opt/gentoo/etc/env.d" CONFIG_SHELL="/opt/gentoo/bin/bash" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/opt/gentoo/var/cache/distfiles" EMERGE_DEFAULT_OPTS=" --autounmask --autounmask-write" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-march=native -O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live candy config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sfpe rms strict unknown-features-warn unmerge-logs unmerge-orphans" FFLAGS="-march=native -O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" MAKEOPTS="-j24" PKGDIR="/opt/gentoo/var/cache/binpkgs" PORTAGE_CONFIGROOT="/opt/gentoo/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/opt/gentoo/var/tmp" RUSTFLAGS="-C target-cpu=native" SHELL="/opt/gentoo/bin/bash" USE="acl amd64 bzip2 caps cli crypt cuda dri eselect-ldso fftw fontconfig fortran gdbm graphite iconv ipv6 jumbo-build libtirpc lto ncurses nls nptl openmp pcre pgo prefix readline seccomp split-usr ssl system-bootstrap system-ffmpeg system-icu system-llvm system-mesa system-wide test-rust unicode zlib zstd" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="abm adx aes sha vaes avx avx2 bmi1 bmi2 fma3 fpu mmx nx sse sse2 sse3 ssse3 sse4_1 sse4_2 f16c cx16 rdseed pclmul" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LANG, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Created attachment 866561 [details] build.log of sci-libs/pytorch-2.0.1
Created attachment 866562 [details] build.log of sci-libs/caffe2-2.0.1-r2
Created attachment 866563 [details, diff] the patch that I use to emerge sci-libs/pytorch
How many files does `find . -type f -name \*.cmake -o -name \*CMakeLists.txt` give? If it is short, let's make an explicit list.
(In reply to Benda Xu from comment #4) > How many files does `find . -type f -name \*.cmake -o -name > \*CMakeLists.txt` give? > > If it is short, let's make an explicit list. I have run ``` grep -ril \ -E "/(usr|lib(|[onx]?32|n?64)|etc|bin|sbin|var|opt|run)" \ $(find . -type f -name \*.cmake -o -name \*CMakeLists.txt) ``` on ${S} of caffe2, and there are 46 files given. I will upload a complete list of outputs.
Created attachment 866645 [details] the CMakeLists.txt and *.cmake files affected by `hprefixify`
(In reply to ZongyuZ from comment #6) > Created attachment 866645 [details] > the CMakeLists.txt and *.cmake files affected by `hprefixify` > ./android/pytorch_android/CMakeLists.txt drop > ./aten/src/ATen/test/CMakeLists.txt > ./aten/src/ATen/CMakeLists.txt > ./aten/CMakeLists.txt keep > ./caffe2/observers/CMakeLists.txt > ./caffe2/CMakeLists.txt keep > ./cmake/External/nccl.cmake > ./cmake/Metal.cmake keep > ./cmake/Modules/FindARM.cmake > ./cmake/Modules/FindAtlas.cmake > ./cmake/Modules/FindBLIS.cmake > ./cmake/Modules/FindFFmpeg.cmake > ./cmake/Modules/FindFlexiBLAS.cmake > ./cmake/Modules/FindHiredis.cmake > ./cmake/Modules/FindLMDB.cmake > ./cmake/Modules/FindLevelDB.cmake > ./cmake/Modules/FindMAGMA.cmake > ./cmake/Modules/FindMKL.cmake > ./cmake/Modules/FindNuma.cmake > ./cmake/Modules/FindOpenBLAS.cmake > ./cmake/Modules/FindOpenMP.cmake > ./cmake/Modules/FindRocksDB.cmake > ./cmake/Modules/FindSnappy.cmake > ./cmake/Modules/FindVSX.cmake > ./cmake/Modules/FindZMQ.cmake > ./cmake/Modules/FindZVECTOR.cmake > ./cmake/Modules/FindvecLib.cmake replace with cmake/Modules/*.cmake > ./cmake/Modules_CUDA_fix/FindCUDNN.cmake > ./cmake/Modules_CUDA_fix/upstream/FindCUDA.cmake > ./cmake/Modules_CUDA_fix/upstream/FindCUDA/make2cmake.cmake > ./cmake/Modules_CUDA_fix/upstream/FindPackageHandleStandardArgs.cmake keep > ./cmake/VulkanCodegen.cmake keep > ./cmake/iOS.cmake drop > ./cmake/public/LoadHIP.cmake > ./cmake/public/cuda.cmake > ./cmake/public/mkl.cmake keep > ./cmake/Dependencies.cmake > ./functorch/CMakeLists.txt > ./test/cpp/api/CMakeLists.txt > ./test/inductor/cpp/CMakeLists.txt keep > ./test/mobile/nnc/CMakeLists.txt drop > ./third_party/nvfuser/CMakeLists.txt > ./torch/lib/libshm/CMakeLists.txt > ./torch/lib/libshm_windows/CMakeLists.txt > ./torch/CMakeLists.txt > ./CMakeLists.txt keep
Update: > ./torch/lib/libshm_windows/CMakeLists.txt drop
Created attachment 866699 [details, diff] explicitly hprefixify the files (In reply to Benda Xu from comment #8) a new patch for caffe2 that explicitly hprefixify the cmake files
LGTM. Tupone and James, please review.
(In reply to Benda Xu from comment #10) > LGTM. Tupone and James, please review. I don't know how to test, but to me, the affected files are only hprefixify \ aten/CMakeLists.txt \ caffe2/CMakeLists.txt \ cmake/Metal.cmake \ cmake/Modules/*.cmake \ cmake/Modules_CUDA_fix/FindCUDNN.cmake \ cmake/Modules_CUDA_fix/upstream/FindCUDA/make2cmake.cmake \ cmake/Modules_CUDA_fix/upstream/FindPackageHandleStandardArgs.cmake \ cmake/public/LoadHIP.cmake \ cmake/public/cuda.cmake \ cmake/Dependencies.cmake \ torch/CMakeLists.txt \ CMakeLists.txt If you can test and is ok, please commit yourself
Created attachment 866759 [details, diff] new patch that affects fewer files (In reply to Tupone Alfredo from comment #11) > (In reply to Benda Xu from comment #10) > > LGTM. Tupone and James, please review. > > If you can test and is ok, please commit yourself I have tried and attached the new patch suggested by Tupone Alfredo. Now caffe2 and pytorch are successfully emerged: ``` [I] sci-libs/caffe2 Available versions: ~1.13.1-r6^t ~2.0.0-r4^t (~)2.0.1-r2^t (~)2.0.1-r2^t[1] (~)2.0.1-r3^t[1] {cuda distributed fbgemm ffmpeg gloo mpi nnpack +numpy opencl opencv openmp qnnpack tensorpipe xnnpack PYTHON_SINGLE_TARGET="python3_10 python3_11"} Installed versions: 2.0.1-r3^t[1](23:35:25 08/01/23)(distributed fbgemm ffmpeg gloo mpi nnpack numpy opencl opencv openmp qnnpack tensorpipe xnnpack -cuda PYTHON_SINGLE_TARGET="python3_11 -python3_10") Homepage: https://pytorch.org/ Description: A deep learning framework [I] sci-libs/pytorch Available versions: ~1.13.1-r2^t ~2.0.0^t (~)2.0.1^t (~)2.0.1^t[1] {debug PYTHON_SINGLE_TARGET="python3_10 python3_11"} Installed versions: 2.0.1^t[1](23:35:41 08/01/23)(-debug PYTHON_SINGLE_TARGET="python3_11 -python3_10") Homepage: https://pytorch.org/ Description: Tensors and Dynamic neural networks in Python [1] "localrepo" /opt/gentoo/var/db/repos/localrepo ```
Created attachment 866760 [details, diff] latest patch (In reply to ZongyuZ from comment #12) > Created attachment 866759 [details, diff] [details, diff] > new patch that affects fewer files > > (In reply to Tupone Alfredo from comment #11) > > (In reply to Benda Xu from comment #10) > > > LGTM. Tupone and James, please review. > > > > If you can test and is ok, please commit yourself > > I have tried and attached the new patch suggested by Tupone Alfredo. I am sorry for uploading a wrong patch, a new version is attached
Created attachment 866761 [details, diff] Obsolete old patches
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=dad0c736cba97bd2cc73f529aa4b10693829a8e9 commit dad0c736cba97bd2cc73f529aa4b10693829a8e9 Author: Alfredo Tupone <tupone@gentoo.org> AuthorDate: 2023-08-01 07:29:25 +0000 Commit: Alfredo Tupone <tupone@gentoo.org> CommitDate: 2023-08-01 07:30:12 +0000 sci-libs/pytorch: prefixify (by ZongyuZ) Closes: https://bugs.gentoo.org/911421 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> sci-libs/pytorch/{pytorch-2.0.1.ebuild => pytorch-2.0.1-r1.ebuild} | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) Additionally, it has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1155091de323b8ccb618a774460e529688b7f748 commit 1155091de323b8ccb618a774460e529688b7f748 Author: Alfredo Tupone <tupone@gentoo.org> AuthorDate: 2023-08-01 07:26:16 +0000 Commit: Alfredo Tupone <tupone@gentoo.org> CommitDate: 2023-08-01 07:30:08 +0000 sci-libs/caffe2: prefixify (by ZongyuZ) Bug: https://bugs.gentoo.org/911421 Signed-off-by: Alfredo Tupone <tupone@gentoo.org> .../{caffe2-2.0.1-r2.ebuild => caffe2-2.0.1-r3.ebuild} | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
