Right now, building for source for other architectures only works reliably for a minimal set of packages and people are resorting to using qemu-user in a chroot with binfmt_misc to cross compile for other architectures. This needs to change so that all Gentoo packages can be cross compiled. So far, I have seen various build failures can be classified into the following categories. Below are descriptions, solutions and examples: 1. Build system builds executes its own build products. The build system must be modified to accept information about the host toolchain and use it to build the build products it executes. If the build products that it executes are also installed, it must build them both for the target architecture and for the host architecture. The tc-getBUILD_*() functions from the toolchain-funcs.eclass can be used to aid with this: https://devmanual.gentoo.org/eclass-reference/toolchain-funcs.eclass/index.html For an example of this failure, cross compile dev-qt/qtcore-5.11.1 and observe it execute its qmake build product. 2. Build system generates library code used by the build system used via CFFI. This is a variation of #1 and the solution is the same. For an example of this failure, cross compile dev-libs/gobject-introspection-1.52.1. 3. Build system does feature detection against host. The solution is to fix the build system to do feature detection against the target inside our EROOT. Autotools checks that rely on running build products must be changed to check for successful compilation. For an example of this failure, cross compile sys-devel/gcc-7.3.0 when the host has dev-util/systemtap installed. 4. Build time dependency generates binary *-config program for external consumers instead of using pkg-config. This does not break the package itself, but it breaks any consumer of the package. The utility either should be replaced by a *.pc file or be rewritten to use a scripting language. For an example of this, cross compile media-libs/mesa-18.1.3 when sys-devel/llvm-6.0.1 is installed on the host. 5. Build system relies on host utility that hard codes version information of host. This is a variation of #4. The ideal solution is to modify the utility to accept information from the build system that replaces the hard coded information and modify the build system to pass it. For an example of this, cross compile dev-qt/qtxml-5.11.1 when dev-qt/qtcore-5.9.4 is installed. 6. Build system includes files from host. This can be from using the host pkgconfig or just bad ./configure options, assuming the build system supports cross compilation. It can also be caused by #4. For an example of this, observe dev-qt/qtcore-5.11.1 compilation.
I feel like this bug report is expecting me. :) I have previously covered most of your points in the cross-boss README as this project aims to deal with a lot of these issues in ways that may be too ugly to fix properly in Gentoo itself. Having said that, since becoming a dev and getting improvements in with EAPI 7, I am more confident that cross-boss will eventually not be needed. I am already making vast changes to it that aren't quite ready yet, hence the lack of commits since 2015. The last major issue is Python paths. I'm close now but was distracted by the arm 17.0 profile migration. https://github.com/chewi/cross-boss
Unfortunately the vast majority of 1-6 boils down to "autotools is broken." The possible solutions are to either 1) build on the target for the target, 2) test against a system which is not the target, or 3) provide static configuration. To explain the situation as best as possible: #1 is a nonstarter as an increasing number of systems running Linux were not meant to self-host, nor will many of those systems likely ever be capable of doing so in any meaningful way. #2 is redundant as duplicating a virtual target as accurately as one would need implies the completion of #3. The project using a build system should probably need to know the impact of the results of a feature test on the codebase. This is ignoring that typical methods of #2 (qemu-user) are often very slow. Static configuration can be emulated already by providing the configuration test answers, but this does not solve the issue of (future) projects which using the broken tests eventually resulting in a failure that needs to be fixed by someone else. Some autotools developers seem to be against supporting static configuration any more than the project already does. Custom build systems are the minority but typically have the same failures as autotools. To solve all of this efficiently the upstream of each affected package would need to participate. When fixing autotools issues it would be wise to see if they can be solved by properly implementing the cross build functionality. Most tests seem to ignore it. Cheers, R0b0t1
(In reply to James Le Cuirot from comment #1) > I feel like this bug report is expecting me. :) I have previously covered > most of your points in the cross-boss README as this project aims to deal > with a lot of these issues in ways that may be too ugly to fix properly in > Gentoo itself. Having said that, since becoming a dev and getting > improvements in with EAPI 7, I am more confident that cross-boss will > eventually not be needed. I am already making vast changes to it that aren't > quite ready yet, hence the lack of commits since 2015. The last major issue > is Python paths. I'm close now but was distracted by the arm 17.0 profile > migration. > > https://github.com/chewi/cross-boss You might like the following patch, although I need to modify the new statx syscall to support more architectures than just aarch64 before I send it upstream: https://paste.pound-python.org/show/OOGzaK2xgQqH09h0ZY38/ (In reply to R030t1 from comment #2) > Unfortunately the vast majority of 1-6 boils down to "autotools is broken." I wouldn't call it autotools being broken. The developers simply used it in a bad way. > Custom build systems are the minority but typically have the same failures > as autotools. All autotools-based build systems are custom. You need to write some code to make it do the right thing for your project. This is where the issues tend to be. > To solve all of this efficiently the upstream of each affected package would > need to participate. When fixing autotools issues it would be wise to see if > they can be solved by properly implementing the cross build functionality. > Most tests seem to ignore it. Many autotools based packages cross compile without an issue. autotools is not the issue here.
I suggest renaming this bug to more explicitly reflect the effort to fix packages and not Gentoo itself. Separate tracker to fix Gentoo's core would make sense as well.
user.eclass need fixing useradd and friends to use --prefix PREFIX_DIR so it works on the cross traget and not on the host.
fixing libtool bug https://bugs.gentoo.org/655820 would be good too
