I'm trying to use the new binary packages support to add back proper support to https://github.com/systemd/mkosi/ for building gentoo images. I'm currently stuck on emerge trying to install various packages to my host system instead of in the image root directory, even when setting $ROOT, $SYSROOT and $BROOT to the root directory of the image. If my understanding is correct, gentoo is trying to install IDEPENDS into the host system root directory instead of the image root directory. Please support installing IDEPENDS (and any other dependency types) into the image root directory so that gentoo images can be built from binary packages from non-gentoo host systems. This could for example be done by adding a new environment variable $IROOT. Reproducible: Always
The point of IDEPEND is basically 'CBUILD RDEPEND' as noted in bug 660306 comment #9. Are you asking to install IDEPEND somewhere else in addition to where the specification says to install them, or to install IDEPEND somewhere else *instead* of where the specification says to install them? If the answer is *instead* then that would mean the specification needs to be changed. If you want to install IDEPEND somewhere else in addition then that's something that could be out of scope of the specification.
(In reply to Daan De Meyer from comment #0) > Please support installing IDEPENDS (and any other dependency types) > into the image root directory so that gentoo images can be built from > binary packages from non-gentoo host systems. Note that if you used gentoo prefix on the non-gentoo host system, then the gentoo prefix is where IDEPEND would be installed, and this would fit within the current specification. https://wiki.gentoo.org/wiki/Project:Prefix
If you use a stage3 chroot approach as I see https://github.com/systemd/mkosi/blob/main/mkosi/distributions/gentoo.py currently has, then maybe you don't need a $ROOT or $EPREFIX since IDEPEND can be installed into the chroot. If you don't want to chroot then gentoo prefix could be an option like I said. Either way, when binary packages are being installed, they need to be able to invoke programs from packages listed in IDEPEND.
I specifically want to move away from the stage3 approach so Gentoo images can be built from e.g. a Fedora system with only portage installed. On a first try, setting EPREFIX seems to affect packages that are installed into ROOT as well. I want EPREFIX to only apply to IDEPENDS. Other package managers install IDEPENDS into ROOT and chroot() into ROOT to execute the postinstall scripts. Can I make emerge do something similar?
My branch with the changes I'm working on to get rid of the stage3 dependency can be found here: https://github.com/DaanDeMeyer/mkosi/blob/gentoo/mkosi/distributions/gentoo.py
(In reply to Daan De Meyer from comment #4) > I specifically want to move away from the stage3 approach so Gentoo images > can be built from e.g. a Fedora system with only portage installed. In theory this should be feasible. However, the stage3 approach is appealing in the sense that it provides greater isolation from the host system, which can make it more robust in practice. > On a first try, setting EPREFIX seems to affect packages that are installed > into ROOT as well. I want EPREFIX to only apply to IDEPENDS. We refer to this as a cross-prefix mode, and we may need to make some adjustments for it work for you. There is an additional PORTAGE_OVERRIDE_EPREFIX that might be useful for cross-prefix scenarios like yours. Portage uses PORTAGE_OVERRIDE_EPREFIX for cross-prefix support internally, like for best_version and has_version here: https://gitweb.gentoo.org/proj/portage.git/commit/?id=17fce85669beec41747d65b37a55747b2f3ddb3c > Other package managers install IDEPENDS into ROOT and chroot() into ROOT to > execute the postinstall scripts. Can I make emerge do something similar? In theory we could implement a chroot mode in portage to do something like this. However, it's tricky to provide the required dependencies for postinstall scripts in the chroot if you're starting from an empty ROOT, so you might need some kind of stage tarball to bootstrap it (this bootstrap should not require build time packages like gcc).
(In reply to Daan De Meyer from comment #4) > Other package managers install IDEPENDS into ROOT and chroot() into ROOT to > execute the postinstall scripts. Can I make emerge do something similar? Indeed they do, which is a neat approach that I've appreciated on those systems. ;) In portage's case, it may be deemed somewhat more complicated due to cross builds, as ROOT may not be the same architecture as the one you are running. For the simple case of e.g. amd64->amd64, a chroot could work... An additional complication is knowing that the ROOT has a functional base-system sufficient to run the things which IDEPEND doesn't mention. e.g. how do you install bash and its dependencies into a blank ROOT, if both packages have to run some scripts?
> In portage's case, it may be deemed somewhat more complicated due to cross builds, as > ROOT may not be the same architecture as the one you are running. For the simple case of > e.g. amd64->amd64, a chroot could work... It's not pretty but user mode emulation tends to work for these cases, even if it is a bit slow. > An additional complication is knowing that the ROOT has a functional base-system > sufficient to run the things which IDEPEND doesn't mention. e.g. how do you install bash > and its dependencies into a blank ROOT, if both packages have to run some scripts? I think most other distributions make sure there's no circular IDEPENDs. Only Debian/Ubuntu stand out, as for these you first have to extract the tarballs of the so called "Essential" packages first, and only then configure them all so that all the required IDEPENDs are available in ROOT. > In theory we could implement a chroot mode in portage to do something like this. However, > it's tricky to provide the required dependencies for postinstall scripts in the chroot if > you're starting from an empty ROOT, so you might need some kind of stage tarball to > bootstrap it (this bootstrap should not require build time packages like gcc). Ideally I wouldn't need a stage tarball at all. If the IDEPENDs could be set up such that there are no circular IDEPENDs, then chroot should theoretically work. I assume that most scripts just need bash, coreutils and extras such as grep, sed, ... . If you make sure these and their dependencies have no IDEPENDs on themselves, chroot()-ing should be possible with an empty ROOT.
(In reply to Zac Medico from comment #1) > If you want to install IDEPEND somewhere else in addition then > that's something that could be out of scope of the specification. (I think see also bug 435066 for that last part.)
(In reply to Daan De Meyer from comment #8) > Ideally I wouldn't need a stage tarball at all. If the IDEPENDs could be set > up such that there are no circular IDEPENDs, then chroot should > theoretically work. I assume that most scripts just need bash, coreutils and > extras such as grep, sed, ... . If you make sure these and their > dependencies have no IDEPENDs on themselves, chroot()-ing should be possible > with an empty ROOT. Gentoo definitely has circular IDEPEND and/or RDEPEND, like the 23 package "cycle" in the attachment to bug 917259 comment #917259 which involves glibc and util-linux. You can imagine that this is only going to get worse and worse over time. What's wrong with using a minimal seed stage for bootstrap anyway? It can be pretty small since it doesn't need to include any build time dependencies.
(In reply to Zac Medico from comment #10) > Gentoo definitely has circular IDEPEND and/or RDEPEND, like the 23 package > "cycle" in the attachment to bug 917259 comment #917259 which involves glibc > and util-linux. I mean attachment 875041 [details].
> Gentoo definitely has circular IDEPEND and/or RDEPEND, like the 23 package "cycle" in the attachment to bug 917259 comment #917259 which involves glibc and util-linux. You can imagine that this is only going to get worse and worse over time. RDEPEND circular dependencies aren't really a problem though, since mkosi would only support binary packages (--usepkgonly). It's only circular IDEPENDs that matter, which seems like a more tractable problem to handle. I only care that packages can be installed without circular dependencies. That they have runtime dependencies on each other isn't a problem. > What's wrong with using a minimal seed stage for bootstrap anyway? It can be pretty small since it doesn't need to include any build time dependencies. It adds extra complexity to mkosi which I'd like to avoid. I want to be able to build gentoo images directly from other distributions without involving an intermediate stage. This works perfectly for all the other distributions we support, so ideally I wouldn't need to make an exception for Gentoo (such an exception would be expected if building from source, but when building from binary packages Gentoo shouldn't need anything extra that other distributions can do without).
(In reply to Daan De Meyer from comment #12) > RDEPEND circular dependencies aren't really a problem though, since mkosi > would only support binary packages (--usepkgonly). It's only circular > IDEPENDs that matter, which seems like a more tractable problem to handle. I > only care that packages can be installed without circular dependencies. That > they have runtime dependencies on each other isn't a problem. We'll have to assume that it's possible to assemble a minimal system using glibc and bash binary packages without running any postinstall scripts, which seems reasonable. We could possibly queue up the postinstall scripts that we didn't run at initial install time, and run them later. Gentoo also supports two types of preinstall scripts via pkg_setup and pkg_preinst functions. Maybe those can invoke those just prior to the queued postinstall scripts, and setup the preinstall script environment as though the packages are being reinstalled. It would be useful to have some way to distinguish essential from non-essential RDEPEND when assembling the minimal system. For example, sys-libs/glibc has an RDEPEND on media-libs/gd which would pull it into the minimal system, but libgd.so.3 is really only used by glibc's /usr/bin/memusagestat which seems non-essential and does not justify pulling media-libs/gd into the minimal system. > > What's wrong with using a minimal seed stage for bootstrap anyway? It can be pretty small since it doesn't need to include any build time dependencies. > > It adds extra complexity to mkosi which I'd like to avoid. I want to be able > to build gentoo images directly from other distributions without involving > an intermediate stage. This works perfectly for all the other distributions > we support, so ideally I wouldn't need to make an exception for Gentoo (such > an exception would be expected if building from source, but when building > from binary packages Gentoo shouldn't need anything extra that other > distributions can do without). It does seem like a nice property to have, and if we queue up preinstall/postinstall scripts and execute them as though packages are being reinstalled, then it only bends our spec a little bit.
Using the current the current binary packages, a command like this results in a minimal system consisting of 25 packages: emerge -pvG --root=/tmp/testroot --ignore-soname-deps=n sys-apps/baselayout app-shells/bash sys-apps/coreutils However, if we add sys-apps/shadow or sys-apps/util-linux to the list then it balloons to more than 300 packages, which seems excessive for a seed stage generated dynamically from binary packages. We could try to optimize the USE flags for this, but then the settings might not be optimal for more general use. If we integrate chroot support into portage, then we can include support for automated download, verification, and extraction of a prebuilt seed stage, so that it doesn't add significant complexity to tools like mkosi.
> However, if we add sys-apps/shadow or sys-apps/util-linux to the list then it balloons to > more than 300 packages, which seems excessive for a seed stage generated dynamically from > binary packages. We could try to optimize the USE flags for this, but then the settings > might not be optimal for more general use. Right this is another problem that Gentoo has when building images. It's very hard to build minimal images without fiddling a lot with USE flags. In mkosi we build initrds out of packages instead of using tools like dracut but with Gentoo this is almost impossible because installing systemd pulls in the kitchen sink (I think ukify is enabled by default which means pulling in python) and changing USE flags means recompiling which we don't want to do. Other package managers solve this by having a concept of subpackages where a single source package is split into multiple binary packages but I don't know if portage supports such a concept.
(In reply to Daan De Meyer from comment #15) > but with Gentoo this is almost impossible because installing systemd pulls > in the kitchen sink (I think ukify is enabled by default which means pulling > in python) and changing USE flags means recompiling which we don't want to > do. No? It is not? Why do you think so? I have changed no USE flags there, and it's not enabled for me. It is even masked on some arches because "sd-boot and ukify are not supported on all archs." (In reply to Daan De Meyer from comment #15) > Other package managers solve this by having a concept of subpackages where a > single source package is split into multiple binary packages but I don't > know if portage supports such a concept. It does not, although I've given Sam my shower thoughts on how/why to do it. You'd basically want to be able to represent it by enabling a USE flag -> rebuild a from-source package or download an additional gpkg.tar, disabling a USE flag -> unmerging part of a package without rebuilding.
