The Firefox build system automatically sets the target architecture when cbindgen calls clang, but it does not handle the sysroot path for finding headers for the target architecture: https://hg.mozilla.org/mozilla-central/file/tip/config/makefiles/rust.mk#l174 Reproducible: Always Steps to Reproduce: 1. cross-emerge -v firefox Actual Results: The build system is very verbose, but here is a failure message. Note that this is compiling from amd64 to arm and the target architecture is correctly set to arm, but the include path contains /usr/include (not the sysroot) and it can't find 32-bit ARM headers. 6:53.19 error: failed to run custom build command for `neqo-crypto v0.4.14 (https://github.com/mozilla/neqo?tag=v0.4.14#ec00592e)` 6:53.19 Caused by: 6:53.19 process didn't exit successfully: `/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/release/build/neqo-crypto-e00e6226ef5a7b4d/build-script-build` (exit code: 101) 6:53.19 --- stdout 6:53.19 cargo:rustc-link-lib=dylib=nssutil3 6:53.19 cargo:rustc-link-lib=dylib=nss3 6:53.19 cargo:rustc-link-lib=dylib=ssl3 6:53.19 cargo:rustc-link-lib=dylib=plds4 6:53.19 cargo:rustc-link-lib=dylib=plc4 6:53.19 cargo:rustc-link-lib=dylib=nspr4 6:53.20 cargo:rustc-link-search=native=/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/bin 6:53.20 cargo:rustc-link-search=native=/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/security/nss/lib/nss/nss_nss3 6:53.20 cargo:rustc-link-search=native=/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/security/nss/lib/ssl/ssl_ssl3 6:53.20 cargo:rustc-link-search=native=/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/config/external/nspr/pr 6:53.20 cargo:rerun-if-changed=/var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/netwerk/socket/neqo/extra-bindgen-flags 6:53.20 cargo:rerun-if-changed=bindings/bindings.toml 6:53.20 cargo:rerun-if-changed=bindings/nspr_time.h 6:53.20 cargo:rerun-if-changed=bindings/nspr_error.h 6:53.20 cargo:rerun-if-changed=bindings/mozpkix.hpp 6:53.20 --- stderr 6:53.20 clang version 11.0.0 6:53.21 Target: armv7a-unknown-linux-gnueabihf 6:53.21 Thread model: posix 6:53.21 InstalledDir: 6:53.21 ignoring nonexistent directory "/usr/local/include" 6:53.21 ignoring nonexistent directory "../../../../lib/clang/11.0.0/include" 6:53.21 ignoring nonexistent directory "/include" 6:53.21 ignoring duplicate directory "/usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr" 6:53.22 ignoring duplicate directory "/usr/include" 6:53.22 #include "..." search starts here: 6:53.22 #include <...> search starts here: 6:53.22 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr 6:53.22 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nss 6:53.22 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss 6:53.22 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/pixman-1 6:53.23 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/cairo 6:53.23 /usr/lib/llvm/11/bin/../../../../lib/clang/11.0.0/include 6:53.23 /usr/include 6:53.23 End of search list. 6:53.23 Failed to run rustfmt: No such file or directory (os error 2) (non-fatal, continuing) 6:53.23 clang version 11.0.0 6:53.24 Target: armv7a-unknown-linux-gnueabihf 6:53.24 Thread model: posix 6:53.24 InstalledDir: 6:53.24 ignoring nonexistent directory "/usr/local/include" 6:53.25 ignoring nonexistent directory "../../../../lib/clang/11.0.0/include" 6:53.25 ignoring nonexistent directory "/include" 6:53.25 ignoring duplicate directory "/usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr" 6:53.25 ignoring duplicate directory "/usr/include" 6:53.25 #include "..." search starts here: 6:53.25 #include <...> search starts here: 6:53.26 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr 6:53.26 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nss 6:53.26 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss 6:53.26 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/pixman-1 6:53.26 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/cairo 6:53.27 /usr/lib/llvm/11/bin/../../../../lib/clang/11.0.0/include 6:53.27 /usr/include 6:53.27 End of search list. 6:53.27 Failed to run rustfmt: No such file or directory (os error 2) (non-fatal, continuing) 6:53.27 clang version 11.0.0 6:53.27 Target: armv7a-unknown-linux-gnueabihf 6:53.28 Thread model: posix 6:53.28 InstalledDir: 6:53.28 ignoring nonexistent directory "/usr/local/include" 6:53.28 ignoring nonexistent directory "../../../../lib/clang/11.0.0/include" 6:53.28 ignoring nonexistent directory "/include" 6:53.28 ignoring duplicate directory "/usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr" 6:53.28 ignoring duplicate directory "/usr/include" 6:53.28 #include "..." search starts here: 6:53.28 #include <...> search starts here: 6:53.28 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr 6:53.28 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nss 6:53.28 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss 6:53.28 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/pixman-1 6:53.28 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/cairo 6:53.28 /usr/lib/llvm/11/bin/../../../../lib/clang/11.0.0/include 6:53.28 /usr/include 6:53.28 End of search list. 6:53.28 /usr/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found 6:53.29 /usr/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found, err: true 6:53.29 thread 'main' panicked at 'unable to generate bindings: ()', /var/tmp/portage/www-client/firefox-84.0.1/work/firefox-84.0.1/third_party/rust/neqo-crypto/build.rs:281:39 6:53.29 stack backtrace: 6:53.29 0: 0x55e2348eae80 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hfb5316c1e3e8df58 6:53.29 1: 0x55e2349103dc - core::fmt::write::hdab8fed0b7ac9734 6:53.29 2: 0x55e2348e79d7 - std::io::Write::write_fmt::heaefdca8f02933f8 6:53.29 3: 0x55e2348ed3d0 - std::panicking::default_hook::{{closure}}::h917202afcdf7c1aa 6:53.29 4: 0x55e2348ed11c - std::panicking::default_hook::h3506873702c2c65b 6:53.29 5: 0x55e2348eda58 - std::panicking::rust_panic_with_hook::hce5a45229bd02ffb 6:53.29 6: 0x55e2348ed609 - std::panicking::begin_panic_handler::{{closure}}::hc323386853bc3a06 6:53.29 7: 0x55e2348eb30c - std::sys_common::backtrace::__rust_end_short_backtrace::hf6fe4bfae53c64c4 6:53.29 8: 0x55e2348ed5c9 - rust_begin_unwind 6:53.29 9: 0x55e23490e9b1 - core::panicking::panic_fmt::h1306a1d4b917e589 6:53.29 10: 0x55e23490e7d3 - core::option::expect_none_failed::h2066faa3e4e1836e 6:53.29 11: 0x55e2343a0bd4 - core::result::Result<T,E>::expect::ha77312ab4187fdc0 6:53.29 12: 0x55e234393d4b - build_script_build::build_bindings::h38aec8e5c64a60f2 6:53.30 13: 0x55e234395ab8 - build_script_build::main::h6407d9c326f3931b 6:53.30 14: 0x55e23439da63 - core::ops::function::FnOnce::call_once::h1c4938447180826b 6:53.30 15: 0x55e23439d949 - std::sys_common::backtrace::__rust_begin_short_backtrace::h18b31d6c0eeaa403 6:53.30 16: 0x55e2343a0179 - std::rt::lang_start::{{closure}}::hdbb2940afbbe37a7 6:53.30 17: 0x55e2348edf01 - std::rt::lang_start_internal::heb360edd22f203b8 6:53.30 18: 0x55e2343a0158 - std::rt::lang_start::hc20ea59f88f32d8a 6:53.30 19: 0x55e23439bf8b - main 6:53.30 20: 0x7f9adb6b7e4b - __libc_start_main 6:53.30 21: 0x55e2343870ca - _start 6:53.30 22: 0x0 - <unknown> 6:53.30 warning: build failed, waiting for other jobs to finish... Expected Results: It should build and install. It supports the BINDGEN_CFLAGS environment variable to specify additional clang options through cbindgen: https://hg.mozilla.org/mozilla-central/file/tip/build/moz.configure/bindgen.configure#l350 I've been setting --sysroot with this in the environment for a while, and it turns the include path into this which fixes it: 7:30.52 [neqo-crypto 0.4.14] #include <...> search starts here: 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nss 7:30.53 [neqo-crypto 0.4.14] /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/include/pixman-1 7:30.53 [neqo-crypto 0.4.14] /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/cairo 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/lib/gcc/armv7a-gentoo-linux-gnueabihf/9.3.0/include/g++-v9 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/lib/gcc/armv7a-gentoo-linux-gnueabihf/9.3.0/include/g++-v9/armv7a-gentoo-linux-gnueabihf 7:30.53 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/lib/gcc/armv7a-gentoo-linux-gnueabihf/9.3.0/include/g++-v9/backward 7:30.54 [neqo-crypto 0.4.14] /usr/lib/llvm/11/bin/../../../../lib/clang/11.0.0/include 7:30.54 [neqo-crypto 0.4.14] /usr/armv7a-gentoo-linux-gnueabihf/usr/include 7:30.54 [neqo-crypto 0.4.14] End of search list. This could be set in the ebuild to fix it for everyone: --- www-client/firefox/firefox-84.0.1.ebuild +++ www-client/firefox/firefox-84.0.1.ebuild @@ -557,6 +557,10 @@ export HOST_CXX="$(tc-getBUILD_CXX)" tc-export CC CXX LD AR NM OBJDUMP RANLIB PKG_CONFIG + # Pass the correct toolchain paths through cbindgen + tc-is-cross-compiler && + export BINDGEN_CFLAGS="${SYSROOT:+--sysroot=${ESYSROOT}} --target=${CHOST} ${BINDGEN_CFLAGS-}" + # Set MOZILLA_FIVE_HOME export MOZILLA_FIVE_HOME="/usr/$(get_libdir)/${PN}"
And just so it's noted, the --target=${CHOST} in the diff does not seem to be redundant. When it's omitted, the search path does not include the GCC header directories, so it is missing standard C++ stuff. 5:59.36 #include <...> search starts here: 5:59.36 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nspr 5:59.36 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/nss 5:59.36 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss 5:59.36 /usr/armv7a-gentoo-linux-gnueabihf/usr/include/pixman-1 5:59.36 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/cairo 5:59.36 /usr/lib/llvm/11/bin/../../../../lib/clang/11.0.0/include 5:59.36 /usr/armv7a-gentoo-linux-gnueabihf/usr/include 5:59.36 End of search list. 5:59.37 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss/mozpkix/pkixtypes.h:28:10: fatal error: 'memory' file not found 5:59.37 /var/tmp/portage/www-client/firefox-84.0.1/work/firefox_build/dist/include/nss/mozpkix/pkixtypes.h:28:10: fatal error: 'memory' file not found, err: true 5:59.37 thread 'main' panicked at 'unable to generate bindings: ()', /var/tmp/portage/www-client/firefox-84.0.1/work/firefox-84.0.1/third_party/rust/neqo-crypto/build.rs:281:39
I've hit this ever since I first tried to cross compile firefox from amd64 to aarch64, and even though I invested half an evening into debugging this more than once I was unable to come up with a solution for it. The fix sounds very promissing, I'm going to test it.
Ping. Test finished?
Created attachment 685536 [details] screenshot from runtime test I did have some success with the ebuild patch, look at configure options to see If you're asking because you want to add the patch to the tree, how about we do that for firefox-86.0 ebuild? This can give a bit more time to runtime test this with armv7a.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=72ff7e264ae4b3f4a5eb5cc4d7c2b825c04c651d commit 72ff7e264ae4b3f4a5eb5cc4d7c2b825c04c651d Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2021-02-05 19:04:18 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2021-02-05 20:02:12 +0000 www-client/firefox: bump to v85.0.1 Bug: https://bugs.gentoo.org/768825 Closes: https://bugs.gentoo.org/761277 Package-Manager: Portage-3.0.14, Repoman-3.0.2 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/Manifest | 97 +++ www-client/firefox/firefox-85.0.1.ebuild | 1137 ++++++++++++++++++++++++++++++ 2 files changed, 1234 insertions(+)
