In Portage, DEPEND currently means "build-time dependencies". This is however insufficient when Portage is used for cross-compiling (using ROOT=...), because build-time dependencies can refer to either the host system, or to the target system. For example, see x11-libs/libxcb-1.5, where DEPEND has the following two entries (which are not in RDEPEND): - app-doc/doxygen, which is host build-time depencency, as it a program needed on the host system during the build, and - x11-proto/xcb-proto, which is a target build-time dependency, as it contains include files that need to be present inside ROOT. Emerge has no way of knowing what kind of a dependency something in DEPEND is. By default, emerge assumes all entries in DEPEND are host build-time dependencies, and installs them to the host if needed. Because of this, cross-compiling x11-libs/libxcb may fail because x11-proto/xcb-proto may not be present in ROOT (but it will be on the host). The default behaviour can be changed with --root-deps: - --root-deps causes emerge to treat build-time dependencies as target dependencies (and install them to ROOT instead of /). This solves the compile failure mention above, but misses host build-time dependencies, and causes circular dependencies when bootstrapping a system. - --root-deps=rdeps ignores all build-time dependencies. This is not really useful when installing packages. To be more useful for cross compilation, it should be possible for ebuilds to seperately list host and target build-time dependencies. I've made a proof-of-concept patch to Portage 2.1.7.17, which allows packages to define the HDEPEND and TDEPEND variables, specifying host and target build-time dependencies, respectively. My modified Portage treats those like that: - if HDEPEND and TDEPEND are empty/undefined, it will act the same way as emerge does now - if any of HDEPEND and TDEPEND are non-empty: - it will treat HDEPEND the same way emerge currently handles DEPEND, that is assume a host dependency and allow to override the behaviour with --root-deps, - it will treat TDEPEND as target build-time dependencies, i.e. the same way as emerge currently treats DEPEND when --root-deps was specified, - it will ignore DEPEND. Also, if an ebuild specifies HDEPEND/TDEPEND, it can remain compatible with portage versions which don't support them by defining DEPEND="${HDEPEND} ${TDEPEND}", assuming all inherited eclasses also support HDEPEND/TDEPEND. Reproducible: Always
Created attachment 229283 [details, diff] proof-of-concept patch
*** This bug has been marked as a duplicate of bug 201499 ***
Created attachment 242403 [details, diff] In EAPI 4, add a TDEPEND variable for build-time target dependencies. The chromium-os people have expressed interest in the ability "to differentiate between host and target build dependencies" (see email thread in URL field), so I've updated your patch as follows: In EAPI 4 or later, DEPEND has been divided into two parts: DEPEND for build-time host dependencies, and TDEPEND for build-time target dependencies. This division is designed specifically to minimize difficulty in the process of adapting ebuilds that were written for earlier EAPIs, and therefore it also minimizes the adjustments that ebuild developers will have to make to the thought processes involved when writing ebuilds from scratch. In an environment that does not involve cross-compilation, TDEPEND behaves the same as DEPEND. When an ebuild is converted from EAPI 3 or earlier to EAPI 4 or later, in order to support cross-compilation environments, some dependencies may need to be migrated to TDEPEND. For ebuilds that have EAPI 4 or later, the emerge --root-deps option has no effect since it is made obsolete by division between DEPEND and TDEPEND. If EAPI 3 or earlier ebuilds are used in combination with EAPI 4 or later ebuilds, the --root-deps behavior will still be applied to the EAPI 3 or earlier ebuilds (there is no behavior change for ebuilds having older EAPIs). The patch applies to the master branch (apply on top of commit cc03d96314dc108964c9c1b0431a0466f20ad147 if it doesn't apply to HEAD) of portage here: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git You can test this patch by creating ebuilds with EAPI=4_pre0_tdepend (this is a temporary EAPI, intended just for testing purposes).
(In reply to comment #3) > Created an attachment (id=242403) [details] > In EAPI 4, add a TDEPEND variable for build-time target dependencies. > I would rather do the more generic solution from bug 201499.
(In reply to comment #4) > I would rather do the more generic solution from bug 201499. That looks pretty nice. However, there may be some trade-offs involved in making a big leap like that (as opposed to making a relatively small incremental change like TDEPEND).
(In reply to comment #5) > > That looks pretty nice. However, there may be some trade-offs involved in > making a big leap like that (as opposed to making a relatively small > incremental change like TDEPEND). > The initial implementation really doesn't need to offer anything new resolver wise. Just a better syntax for declaring the current things.
Created attachment 242691 [details, diff] In EAPI 4, add HDEPEND variable for build-time target dependencies. Based on feedback from chromium-os people (see email thread in URL field), I've adjusted the previous TDEPEND patch into an HDEPEND patch. This patch applies on top of commit ffe0e3fb6998c776b08f725c53dfe6c632ce7d02 of the master branch. You can test it by creating ebuilds with EAPI=4_pre0_hdepend (this is a temporary EAPI, intended just for testing purposes). (In reply to comment #6) > The initial implementation really doesn't need to offer anything new resolver > wise. Just a better syntax for declaring the current things. Well, this HDEPEND patch is small enough and complete such that we should have no problem working it into EAPI 4. If feedback from the chromium-os people is positive, I plan to write up a short glep and post it on the gentoo-dev ml.
People will need to rewrite their dependencies and change their eclasses for this anyway. Why not move to a labels based solution rather than introducing yet more not entirely obviously based names?
(In reply to comment #8) > People will need to rewrite their dependencies and change their eclasses for > this anyway. Why not move to a labels based solution rather than introducing > yet more not entirely obviously based names? GLEP it. this applies for both, mangling dep strings like this is something that requires community wide feedback.
Reopening, since the HDEPEND approach differs from the exheres / unified DEPENDENCIES approach discussed in bug 201499.
*** Bug 263343 has been marked as a duplicate of this bug. ***
Until this is implemented, could we please install DEPEND into both host *and* target? That might install a bit more than necessary, but would at least allow crosscompilation. For example sys-kernel/dracut-023 (DEPEND="app-text/asciidoc") currently fails to crosscompile if I did not manually install app-text/asciidoc into the hostsystem, because that is where it tries to run asciidoc from.
(In reply to comment #12) > Until this is implemented, could we please install DEPEND into both host > *and* target? That might install a bit more than necessary, but would at > least allow crosscompilation. You can get similar results by using --root-deps and in combination with a hard-host-deps ebuild like they use for chromium-os: http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build-faq > For example sys-kernel/dracut-023 (DEPEND="app-text/asciidoc") currently > fails to crosscompile if I did not manually install app-text/asciidoc into > the hostsystem, because that is where it tries to run asciidoc from. So, you can create a hard-host-deps ebuild with RDEPEND="app-text/asciidoc". Alternatively, I suppose we could add a --host-deps option that you'd be able to use together with --root-deps to get the behavior that you've requested.
(In reply to comment #13) > Alternatively, I suppose we could add a --host-deps option that you'd be > able to use together with --root-deps to get the behavior that you've > requested. Please go ahead and file a bug for dev-portage@gentoo.org if this interests you.
I find this reasoning very strange.... The HDEPEND solves this bug, for the majority of cases, probably. But it isn't perfect (like everything else), so yes, there are a small number of cases where it would not work right (version mismatch between host and target, to be specific, see http://comments.gmane.org/gmane.linux.gentoo.devel/79700 ). Because of this hypothetical issue, which I could not even find demonstrated in practice, the entire idea seems to have been dropped. Instead, a clearly inferior solution may be developed (and it's really a solution). What happened to incremental improvement? Why not just try it, see where it leads, and fix stuff when actual problems come up? To me this certainly seems better than doing nothing about it. Portage has never been perfect. There will always be things it can't handle. Just look for packages which do runtime check of USE flags for example. There's nothing preventing a package from doing a runtime check if the host/target versions don't match but should.
The primary problem is that as far as we can work out, to express dependencies correctly (as opposed to just throwing in enough to make it seem to work with one particular resolver), nearly every DEPEND-but-not-RDEPEND would have to be converted to an HDEPEND. Thus it may be better to make DEPEND do what you think HDEPEND should do, and then see if there are any legitimate use cases for adding a TDEPEND to do what DEPEND currently does. Just adding an HDEPEND "to make a few things appear to work" causes more problems than it solves.
I've taken a look at some random packages, and it does seem that most of the DEPEND-but-not-RDEPEND are host-only dependencies. So I agree, it would make sense to have DEPEND mean host dependencies and TDEPEND target build-time dependencies - less packages would need to be fixed, and the changes would be smaller. > and then see if there are any legitimate use cases for adding a TDEPEND to do what DEPEND currently does. My prediction is a package will very soon pop up which needs it, and then it will take another EAPI generation or two to fix it. It's a very small thing and easy to implement it - what damage could it possibly do? So, does someone see any other showstopper problems with this? And please, don't say stuff like "we should do X, it would look a bit nicer, and would take N years for someone to actually implement it". I would really like this bug to be solved at once. It would make Gentoo so much more practical for embedded systems. Zac, since you've been taking care of this patch, do you think you could change it to do TDEPEND (assuming others agree that's the way to go)?
(In reply to comment #17) > > and then see if there are any legitimate use cases for adding a TDEPEND to do what DEPEND currently does. > My prediction is a package will very soon pop up which needs it, and then it > will take another EAPI generation or two to fix it. It's a very small thing > and easy to implement it - what damage could it possibly do? We don't know that there are any legitimate reasons to use it. Adding it would make people think that there are such reasons, so they might use it when it's not needed. Incidentally, the "no legitimate reasons" thing *does* depend upon us getting IDEPEND. > So, does someone see any other showstopper problems with this? And please, > don't say stuff like "we should do X, it would look a bit nicer, and would > take N years for someone to actually implement it". I would really like this > bug to be solved at once. It would make Gentoo so much more practical for > embedded systems. Won't just changing DEPEND to do what you think HDEPEND should do do that without the need for a new variable or any package changes?
> We don't know that there are any legitimate reasons to use it. > Adding it would make people think that there are such reasons, > so they might use it when it's not needed. Take a look at x11-libs/libxcb DEPEND'ing on x11-proto/xcb-proto. That's a pure TDEPEND. You need it at build time in the target root, and not at runtime. Without TDEPEND this would either be kept in DEPEND and may fail to cross compile, or it would be moved to RDEPEND, wasting space. (btw I mentioned that package in the original report :) I think TDEPEND would also be applicable to applications which link statically to libraries (assuming they don't need other parts of the package at runtime). Yes, most of the time static linking should be avoided, but sometimes it's still useful. Think e.g. critical components like stuff needed to boot and compilers, and how it could badly mess up your system if a library dependency got broken after update. (yes, I know about preserve-libs and the "libraries have not been removed" thing) > Won't just changing DEPEND to do what you think HDEPEND should do do that without the need for a new variable or any package changes? As said, not for some packages. There are probably more like the libxcb case. Probably rare, but they do exist. > Incidentally, the "no legitimate reasons" thing *does* depend upon > us getting IDEPEND. I don't know much about IDEPEND, at first lookit seems orthogonal to the this problem.
(In reply to comment #18) > Incidentally, the "no legitimate reasons" thing *does* depend upon us > getting IDEPEND. What is IDEPEND? > > So, does someone see any other showstopper problems with this? And please, > > don't say stuff like "we should do X, it would look a bit nicer, and would > > take N years for someone to actually implement it". I would really like this > > bug to be solved at once. It would make Gentoo so much more practical for > > embedded systems. > > Won't just changing DEPEND to do what you think HDEPEND should do do that > without the need for a new variable or any package changes? HDEPEND is supposed to contain DEPEND atoms that need to be present in /? That is what I always assumed DEPEND means. So, yes, I agree that changing DEPEND is a good idea! However, what about stuff like header-only libraries (I am e.g. thinking of dev-cpp/eigen atm)? They need to be installed in ROOT, right? And still they are build-time-only dependencies. Hence my suggestion to install DEPEND into ROOT as well as /, and be done with it. See bug #435066. If someone wants to have a minimal system for embedded use (i.e. without DEPEND), they can build binary packages from ROOT and install them whereever they want (e.g. by calling portage with a different ROOT and --usepkgonly).
IDEPEND is for the generalisation of header-only libraries. It's for dependencies that are required when a package is used to satisfy a DEPEND, but not for when a package is used to satisfy an RDEPEND.
Uh, everything I said about IDEPEND should say BADEPEND. Also, we really need longer variable names...
> However, what about stuff like header-only libraries (I am e.g. > thinking of dev-cpp/eigen atm)? They need to be installed in ROOT, > right? And still they are build-time-only dependencies. This is also a good case for TDEPEND - needed in ROOT at compile time. > Hence my suggestion to install DEPEND into ROOT as well as /, and > be done with it. See bug #435066. That would make the already cumbersome cross-compilation process even more so. Especially for a package which doesn't work on the host, or conflicts with existing packages there.
(In reply to comment #22) > Uh, everything I said about IDEPEND should say BADEPEND. > > Also, we really need longer variable names... I found a page in the wiki [1], and must say it's a mess. Everybody and their mom gets their own DEPEND variable. I do not expect ebuild writers to memorize all of that. [1] http://wiki.gentoo.org/wiki/Future_EAPI/New_dependency_types (In reply to comment #23) > > However, what about stuff like header-only libraries (I am e.g. > > thinking of dev-cpp/eigen atm)? They need to be installed in ROOT, > > right? And still they are build-time-only dependencies. > This is also a good case for TDEPEND - needed in ROOT at compile time. > > > Hence my suggestion to install DEPEND into ROOT as well as /, and > > be done with it. See bug #435066. > That would make the already cumbersome cross-compilation process even more > so. Especially for a package which doesn't work on the host, or conflicts > with existing packages there. You mean a package that might not even be possible to build for the host? Maybe due to missing keywords? Maybe something like sys-libs/libspe2, that makes no sense at all on a non ppc (or non IBM Cell, to be specific) system? In that case the flags (--install-*depend=) suggested in bug #435066 would at least ease the situation. A complete solution that would be acceptable to maintain by ebuild writers appears rather complex…
> In that case the flags (--install-*depend=) suggested in bug #435066 would > A complete solution that would be acceptable to maintain by ebuild writers > appears rather complex. Put it into TDEPEND. How is that complex? > I do not expect ebuild writers to memorize all of that. So, in other words, you don't want this bug solved? You'd rather apply a hack that works half of the time and doesn't the other half, just because adding this new support means that someone will have to use it properly? AFAIK, ebuilds are not required to cross-compile to be accepted into the tree. For those people who do want to get an ebuild to cross-compile, TDEPEND will be a very useful feature. Others are free not to care, saving bits in their brain :)
(In reply to comment #17) > Zac, since you've been taking care of this patch, do you think you could > change it to do TDEPEND (assuming others agree that's the way to go)? There's already a TDEPEND patch attached to this bug, which you can see if you click the "Show Obsolete" attachments link. I really don't care whether we split out TDEPEND or HDEPEND, but the dominant preference expressed on the chromium-os-dev mailing list was to split out HDEPEND, since they felt that it would result in fewer changes to the dependencies of existing ebuilds: http://groups.google.com/a/chromium.org/group/chromium-os-dev/browse_thread/thread/c9501ca5f647ace1
(In reply to comment #25) > > I do not expect ebuild writers to memorize all of that. > So, in other words, you don't want this bug solved? I want the underlying issue be solved. That includes a way to workaround packages which currently do not automatically do the right thing. And I want to take care that ebuild writing does not get more complex by default. > AFAIK, ebuilds are not required to cross-compile to be accepted into the > tree. For those people who do want to get an ebuild to cross-compile, > TDEPEND will be a very useful feature. Others are free not to care, saving > bits in their brain :) Having had a look at that wiki page I mentioned, I see feature creep written all over it. What I fear is, that RDEPEND, TDEPEND, HDEPEND, IDEPEND, BADEPEND, ... (you get the picture) get shoved into the face of the average developer (or worse: contributor), and knock them off their feet. IMO it's enough work already to write a clean ebuild, and get it accepted. (Basically what ferringb said in the chromium-os thread zmedico linked to.) So I just got a bit scared by what people are apparently trying to do to ebuild writing. Coming back to the original issue: There needs to be a simple, intuitive and easily described meaning to the core variables. And they should be few. Their meaning should not change if other, additional variables are used. Then there should be purely optional variables, to deal with cases like the original issue. That said, I second what Zac said in comment #26 about splitting out HDEPEND. Then RDEPEND and DEPEND purely affect ROOT, and their meaning stays simple and consistent. If an ebuild turns out to need something in addition, HDEPEND can be used. TDEPEND can be saved - one variable less.
(In reply to comment #27) Some idea I draw from ferringb's post over at chromium-os: We have sandbox - can that be used to automatically gather information about what tools are being called and what packages they belong to? That would allow ebuild writers to run a test build and see what HDEPEND portage suggests. While I am at it: Maybe the produced executables could also be examined, to aid in setting RDEPEND? It could not be followed blindly, of course, but help categorising the dependencies.
Considering the "HDEPEND or TDEPEND" problem: IMO adding TDEPEND is best option for Gentoo. While the Chromium OS people might prefer HDEPEND, they will have to redo all their dependencies anyway, so this is probably not a big deal. In Gentoo however, many packages can just stay the same with the TDEPEND scheme. This includes already cross-compilable packages where the DEPEND is just build tools (most DEPENDs are), and packages which are *not* cross-compilable (most of them). If, on the other hand, HDEPEND was used, those not-cross-compilable (most) packages would all have to rename DEPEND->HDEPEND when porting to new EAPI. (In reply to comment #28) > (In reply to comment #27) > Some idea I draw from ferringb's post over at chromium-os: We have sandbox - > can that be used to automatically gather information about what tools are > being called and what packages they belong to? That would allow ebuild > writers to run a test build and see what HDEPEND portage suggests. While I > am at it: Maybe the produced executables could also be examined, to aid in > setting RDEPEND? It could not be followed blindly, of course, but help > categorising the dependencies. Maybe, but considering the amount of programs that get called, this may not be worth much. I think it's enough if people just report bugs and stuff will get fixed. After all, it's just about moving something to TDEPEND (and possibly keeping it in DEPEND). For best efficiency, the ebuild can be ported to the new EAPI along the way.
> If, on the other hand, HDEPEND was used, those not-cross-compilable > (most) packages would all have to rename DEPEND->HDEPEND when porting > to new EAPI. Ah, sorry, not entirely true. Those could keep using DEPEND, since DEPEND and HDEPEND would have the same meaning for native compile. But the "more DEPENDs are host depends than target depends" still applies - with TDEPEND, less packages would have to be fixed to get them to cross-compile, because it's more likely their dependencies would turn out right if using TDEPEND as opposed to HDEPEND.
What I see is, that there are a lot more library and header dependencies (that are dealt with by the cross-toolchain) than tool dependencies (that actually need to be executed during build). So TDEPEND would contain a lot more than DEPEND. Hence it seems to make sense to use HDEPEND and make --root-deps the default behaviour. I.e. DEPEND goes into ROOT and HDEPEND specifies extra tools that need to be executed during the build. I would then also remove --root-deps, as it would become plain confusing.
> What I see is, that there are a lot more library and header dependencies > (that are dealt with by the cross-toolchain) than tool dependencies (that > actually need to be executed during build). So TDEPEND would contain a > lot more than DEPEND. Ah, I get your point. DEPEND currently needs to include tools *and* libraries so that is it guaranteed they will be present at build time. So we generally have two types of packages which need to be considered. The simple ones currently have DEPEND==RDEPEND==libraries: DEPEND="libraries" RDEPEND="${DEPEND}" (or the other way) And more complex ones have extra build-only tools or run-only tools: (yes it can be simpler) COMMON_DEPEND="libraries" DEPEND="${COMMON_DEPEND} build tools X11 or other headers" RDEPEND="${COMMON_DEPEND} runtime programs" Now what happens if we implement DEPEND = host dependencies, TDEPEND = target build dependencies? - The first kind of packages will have to be fixed to rename DEPEND to TDEPEND. - The second kind will have to be fixed, possibly like that: COMMON_DEPEND="libraries" TDEPEND="${COMMON_DEPEND} X11 or other headers" RDEPEND="${COMMON_DEPEND} runtime programs" DEPEND="build tools" But, if we use DEPEND = target build time and add HDEPEND = host, all those simple packages will remain correct! (complex one will need fixing anyway)
I'm now convinced that adding HDEPEND is the way to go. Thank you for pointing that out, Dennis. I'll try applying/updating the current patch and get some cross-compilation going.
Haaang on. Why don't you look at a bunch of packages and explicitly classify where every dependency has to be and when it has to be there? We've still not seen a single conclusive instance of a dependency that isn't covered by existing proposals.
> We've still not seen a single conclusive instance of a dependency that > isn't covered by existing proposals. What do you mean? The x11-proto stuff is covered by both TDEPEND and HDEPEND, but cannot be solved without them (did you have some other proposal in mind)? If we did not add a new variable, but instead only changed DEPEND to mean either host or target, it won't work. If DEPEND was defined to mean host dependencies, consider all such packakes: RDEPEND="libraries" DEPEND="${RDEPEND} some host tools" these would suddenly have to install lots of unneeded libraries to the host, and cause problems when that doesn't work (blockers, build failures, non-applicable packages). On the other hand, if DEPEND was defined to mean target dependencies, there would be no way to depend on host tools. And, if it was defined to mean *both* host and target dependencies, you'd get those build tools, but have lots of unneeded packages on the host and the problems just mentioned above. To sum up, TDEPEND or HDEPEND is needed (or an equivalent feature, but I strongly prefer those, because they're already implemented, small and non-intrusive).
> If DEPEND was defined to mean host dependencies plus the fact that libraries may not be present at build time
(In reply to comment #35) > The x11-proto stuff is covered by both TDEPEND and HDEPEND, but cannot be > solved without them (did you have some other proposal in mind)? That's what's generally known as a BADEPEND. Headers-only libraries are not a cross-compile- or ROOT- specific problem.
Yes, but what about build tools? The problem still stands: you want to install on the host only those packages that are truly needed (build tools), and not every single thing in DEPEND (lots of stuff).
> Haaang on. Why don't you look at a bunch of packages and explicitly > classify where every dependency has to be and when it has to be there? sys-libs/ldb becomes: RDEPEND="dev-libs/popt >=sys-libs/talloc-2.0.7[python] >=sys-libs/tevent-0.9.16 >=sys-libs/tdb-1.2.9[python] net-nds/openldap !!<net-fs/samba-3.6.0[ldb] !!>=net-fs/samba-4.0.0[ldb] " DEPEND="${RDEPEND}" HDEPEND="dev-libs/libxslt doc? ( app-doc/doxygen ) virtual/pkgconfig" sys-libs/libsemanage becomes: RDEPEND=">=sys-libs/libsepol-${SEPOL_VER} >=sys-libs/libselinux-${SELNX_VER} dev-libs/ustr ruby? ( dev-lang/ruby )" DEPEND="${RDEPEND}" HDEPEND="sys-devel/bison sys-devel/flex ruby? ( >=dev-lang/swig-2.0.4-r1 ) python? ( >=dev-lang/swig-2.0.4-r1 )" sys-apps/util-linux becomes: RDEPEND="!sys-process/schedutils !sys-apps/setarch !<sys-apps/sysvinit-2.88-r4 !sys-block/eject !<sys-libs/e2fsprogs-libs-1.41.8 !<sys-fs/e2fsprogs-1.41.8 cramfs? ( sys-libs/zlib ) ncurses? ( >=sys-libs/ncurses-5.2-r2 ) perl? ( dev-lang/perl ) selinux? ( sys-libs/libselinux ) slang? ( sys-libs/slang ) udev? ( sys-fs/udev )" DEPEND="${RDEPEND} virtual/os-headers" HDEPEND="nls? ( sys-devel/gettext )" Actually, many of those blockers could probably be moved out of DEPEND, if they're not blocking the build itself, but only installation of the package. Of course all this is just a guess, the maintainers would know better.
Build tools are pure DEPENDs, which means they belong on /. Libraries are in both DEPEND and RDEPEND, which means they belong on both / and in the ROOT.
> Libraries are in both DEPEND and RDEPEND, which means they belong on both > / and in the ROOT. Which is the whole point. You don't usually need a library to be installed in / in order to link against a version of this library in ROOT. The library might not even be buildable for the host (e.g. it's architecture-specific) or it may conflict with existing packages in /.
(In reply to comment #41) > > Libraries are in both DEPEND and RDEPEND, which means they belong on both > > / and in the ROOT. > Which is the whole point. You don't usually need a library to be installed > in / in order to link against a version of this library in ROOT. The library > might not even be buildable for the host (e.g. it's architecture-specific) > or it may conflict with existing packages in /. You're trying to complicate things by considering the cross-compile case and the ROOT case to be the same thing: "/" and "native" are distinct concepts. We need to get ROOT right before we even start to look at cross-compiling.
> You're trying to complicate things by considering the cross-compile case > and the ROOT case to be the same thing: "/" and "native" are distinct > concepts. We need to get ROOT right before we even start to look at > cross-compiling. I'm very afraid from your argument that because native ROOT will not be "gotten right" for a long time (if ever), this will stall cross-compiling support. And as far as I see it, with HDEPEND, you get good cross-compiling capability, right now. If someone find a nice way to get native ROOT right, I don't see how it couldn't be implemented after HDEPEND exists (or converse, why implementing cross compiling would be easier once native ROOT work right).
That being said, it would of course be good if any new features didn't make native ROOT any worse. Is there some docs which describes the current support for native ROOT?
Branching off the discussion and throwing something a bit random: Somewhere in the posts on the matter there was the mention of the BSD ports variable LIB_DEPEND. Now that is a nice name. :) If we throw in BUILDTOOL_DEPEND (stuff that needs to be executed during build) and BUILDLIB_DEPEND (libraries/headers only needed during build), I think we have nice names that everyone immediately understands, and that cover our problem. No non-cross-compilers would have to think about the concept of host vs target, ROOT vs / — they just check whether it is a tool to be run during build, or a library/header/… to be used and put it in the appropriate place. Portage can then figure out where to install them — be it the host or the target, ROOT or /. (In reply to comment #42) > You're trying to complicate things by considering the cross-compile case and > the ROOT case to be the same thing: "/" and "native" are distinct concepts. > We need to get ROOT right before we even start to look at cross-compiling. Can you please elaborate on how / would not mean "native"? Where are your executable build tools located, if not in /? (In reply to comment #40) > Build tools are pure DEPENDs, which means they belong on /. Agreed. How to name the var is still up to discussion. > Libraries are in both DEPEND and RDEPEND, which means they belong on both / > and in the ROOT. Disagreed. S.a. — there might be libraries which cannot be installed into /. I do not even see how they are needed there. Linking against them happens relative to ROOT, not within the environment of /. Same goes for headers, pkg-config and such. They always belong into ROOT, though — I agree there.
(In reply to comment #43) > I'm very afraid from your argument that because native ROOT will not be > "gotten right" for a long time (if ever), this will stall cross-compiling > support. And as far as I see it, with HDEPEND, you get good cross-compiling > capability, right now. If someone find a nice way to get native ROOT right, > I don't see how it couldn't be implemented after HDEPEND exists (or > converse, why implementing cross compiling would be easier once native ROOT > work right). The cross-compiling problem is a superset of getting ROOT right, though. With HDEPEND, you're looking at adding in a hack that will do just enough that it will appear to work in certain specific situations until the resolver changes slightly. That doesn't really solve anything, and just creates lots of work in the long term. Fixing this comes down to classifying dependencies *properly*. (In reply to comment #44) > That being said, it would of course be good if any new features didn't make > native ROOT any worse. Is there some docs which describes the current > support for native ROOT? From a dependencies perspective, it pretty much doesn't exist. DEPEND and RDEPEND simply aren't sufficient. That's one of the things the "huge list of *DEPEND or DEPENDENCIES" proposals start to solve. What you need to do is find what cross compiling needs on top of that, if anything -- not to hack things to appear to work, but to express dependencies correctly.
(In reply to comment #44) > That being said, it would of course be good if any new features didn't make > native ROOT any worse. Is there some docs which describes the current > support for native ROOT? Currently (at least back when I wrote bug #375751), if ROOT is the same arch as /, DEPEND and RDEPEND appear to end up in ROOT, but also in /. (In reply to comment #43) > > You're trying to complicate things by considering the cross-compile case > > and the ROOT case to be the same thing: "/" and "native" are distinct > > concepts. We need to get ROOT right before we even start to look at > > cross-compiling. > > I'm very afraid from your argument that because native ROOT will not be > "gotten right" for a long time (if ever), this will stall cross-compiling > support. And as far as I see it, with HDEPEND, you get good cross-compiling > capability, right now. If someone find a nice way to get native ROOT right, > I don't see how it couldn't be implemented after HDEPEND exists (or > converse, why implementing cross compiling would be easier once native ROOT > work right). I do not see at all how HDEPEND (or BUILD*_DEPEND, s.a.) would hurt native ROOT at all. HDEPEND goes into /, because it specifies tools need to be run during build. It should not matter whether you execute the tool from ROOT or /. Another random thought regarding HDEPEND/BUILDTOOL_DEPEND: These would need to support cross-compilation in many cases. Or at least use wrappers that point them into ROOT instead of /. See for example pkg-config, but also cmake or similar tools that need to look for target-specific configurations. Not sure how to do that without writing cross-$CTARGET/pkgname everywhere…
> Somewhere in the posts on the matter there was the mention of the BSD > ports variable LIB_DEPEND. Now that is a nice name. :) If we throw in > BUILDTOOL_DEPEND (stuff that needs to be executed during build) and > BUILDLIB_DEPEND (libraries/headers only needed during build), I think > we have nice names that everyone immediately understands, and that > cover our problem. I'm all for making names more descriprive. So, you'd have: - LIB_DEPEND. Needed in ROOT both during build and at runtime. (e.g. shared libraries) - BUILDLIB_DEPEND. Needed in ROOT during build. (e.g. header-only libraries) - BUILDTOOL_DEPEND. Needed in / during build. One part's missing however - stuff needed in ROOT for the package to actually work at runtime, but not needed to build it. Like programs this package calls. Maybe: - RUN_DEPEND. Needed in ROOT before package is installed (including from binary). If this is also needed in ROOT at build time, LIB_DEPEND should be used *instead*. But the x11-proto problem still remains. That is, it is not enough for x11-proto/P1 to be in BUILDLIB_DEPEND. The problem is that the P1 headers may #include x11-proto/P2 headers. Solution for this was already suggested: - BUILD_AGAINST_DEPEND - Required in ROOT when this package is pulled in via a BUILDLIB_DEPEND, or recursively via BUILD_AGAINST_DEPEND. Now, can anyone see a case not covered by these?
> - BUILD_AGAINST_DEPEND - Required in ROOT when this package is pulled > in via a BUILDLIB_DEPEND, or recursively via BUILD_AGAINST_DEPEND. Probably also when pulled in via LIB_DEPEND.
> With HDEPEND, you're looking at adding in a hack that will do just enough > that it will appear to work in certain specific situations until the > resolver changes slightly. Why exactly do you consider this a hack? What would break? HDEPEND just says you need something available on the host system at build time, to be run during the build process. Are you concerned about version problems? Use version-based dependencies. To be very particular, consider that package X needs package X (same one) at runtime (which happens from time to time, I think GTK+ is like that). If it needs a build tool from package X of the same version, it can say stuff like - HDEPEND="${P}-${PV}" - HDEPEND="~${P}-${PV}" - HDEPEND=">=${P}-${PV}" - HDEPEND=">=${P}-3 <${P}-4" or whatever it really needs.
> needs package X (same one) at runtime Sorry that should say "at build time".
(In reply to comment #48) > But the x11-proto problem still remains. That is, it is not enough for > x11-proto/P1 to be in BUILDLIB_DEPEND. The problem is that the P1 headers > may #include x11-proto/P2 headers. Solution for this was already suggested: > - BUILD_AGAINST_DEPEND - Required in ROOT when this package is pulled in via > a BUILDLIB_DEPEND, or recursively via BUILD_AGAINST_DEPEND. You mean like the stuff libtool does using .la files, or pkg-config does for building against static libs? One reason why H/TDEPEND var names are misleading when used as synonyms for / and ROOT dependencies: With native ROOT, there is no cross-compiler. Hence headers need to be in /, because there is no easy way of telling the compiler to use a different default include-path (afaik). So TDEPEND would probably break that, because the deps would only go into ROOT. Or portage would decide to put them into / and the var name would suddenly be wrong. This is IMO one more reason to use descriptive "what-this-is-used-as" names (i.e. BUILD*/LIB_DEPEND), instead of "where-should-this-go" dependencies like H/TDEPEND. I am not sure about the names LIB_ and RUN_DEPEND. dlopened libraries are libs and still need to be in RUN_ instead of LIB_. Then I do not see why we would need both LIB_ and RUN_DEPEND. In ebuilds you could write: COMMON_DEPEND="…" BUILDTOOL_DEPEND="…" BUILDLIB_DEPEND="${COMMON_DEPEND} …" RUN_DEPEND="${COMMON_DEPEND} …" and achieve the same effect with less variables. RUN_DEPEND would mean anything (library or executable) needed to run the program. (I.e. the program that is the result of the package.) If you want, you can even rename that stuff a bit and arrive with less descriptive but more traditional names: DEPEND <- BUILDLIB_DEPEND BDEPEND <- BUILDTOOL_DEPEND RDEPEND <- RUN_DEPEND The old EAPI would be treated as: DEPEND="${BUILDLIB_DEPEND} ${BUILDTOOL_DEPEND}" RDEPEND="${RUN_DEPEND}" i.e. BDEPEND would have been separated from DEPEND.
I think the exact role of dependencies could do with some clarification. For ROOT dependencies, we need a two dimensional matrix, like this: V When it's needed Where it's needed --> On / In ROOT At fetch time At compile time At install time When used as a compile dep When used as a dep As a goal At some point Some of the cells can be empty. It's not entirely clear how this gets filled in currently for Gentoo dependencies, and the spec doesn't really say. Paludis does it like this: V When it's needed Where it's needed --> On / In ROOT At fetch time At compile time DEPEND At install time (DEPEND intersect RDEPEND) When used as a compile dep When used as a dep RDEPEND As a goal RDEPEND At some point PDEPEND What does Portage do, and what should it do? For exheres-0, and probably the full-blown version of the DEPENDENCIES proposal, it looks like this: V When it's needed Where it's needed --> On / In ROOT At fetch time fetch At compile time build At install time install When used as a compile dep built-against When used as a dep run As a goal run At some point post (exheres-0 also has 'test', which Gentoo does via a USE flag instead, and suggestions and recommendations, which are like 'post'.) For cross-compiling, we either need a third dimension, for native and target, or to double the number of columns. So how would this be filled in, and which cells have legitimate use cases? V When it's needed Where it's needed --> On / native On / cross In ROOT native In ROOT cross At fetch time At compile time At install time When used as a compile dep When used as a dep As a goal At some point
(In reply to comment #53) > When used as a compile dep > When used as a dep What are these two? > As a goal What is a "goal"? > For cross-compiling, we either need a third dimension, for native and > target, or to double the number of columns. So how would this be filled in, > and which cells have legitimate use cases? What is definitely necessary is to differentiate between compile-time-tools and compile-time-libs(/headers/…). The former need to go into / and the latter into ROOT, when cross-compiling. When doing a native build for ROOT, everything just goes into /. (In reply to comment #53) > On / native On / cross In ROOT native In ROOT cross When is / ever cross-compiled? Instead of using a 3 dimensional matrix, why don't you just use 2 matrices, one for each usecase (native/foreign ROOT)? Someone still has to read (and understand) that…
(In reply to comment #54) > (In reply to comment #53) > > When used as a compile dep > > When used as a dep > What are these two? The first is a generalisation of a "header only library". If a package #includes headers from a header-only library, then the header-only library needs to be installed. This isn't a build dependency, since build dependencies can be removed after the depending package has been installed. It's also not a run dependency, since it's not required at runtime. The second is "things that a package needs to be usable". > > As a goal > What is a "goal"? A package you actually care about, as opposed to one that's just there to let you install a goal. > When is / ever cross-compiled? If you think the answer is "never", then the entire column can be empty, and you can omit it. That leaves you with a three column matrix to fill in.
*** Bug 431084 has been marked as a duplicate of this bug. ***
(In reply to comment #53) > For exheres-0, and probably the full-blown version of the DEPENDENCIES > proposal Is exheres-0 going into Gentoo? A quick googling brought me to Exherbo [1] and it appears like exheres-0 has nothing to do with Gentoo anymore, but with another distro, which is just an idea-fork of Gentoo and Portage? Hence I think we do not need to talk about it here. Please correct me if I am wrong. [1] http://exherbo.org/docs/exheres-for-smarties.html (In reply to comment #55) > The first is a generalisation of a "header only library". If a package > #includes headers from a header-only library, then the header-only library > needs to be installed. This isn't a build dependency, since build > dependencies can be removed after the depending package has been installed. > It's also not a run dependency, since it's not required at runtime. So basically the same concept as libtool .la files or pkg-config's Libs — indirect header dependencies and static libraries? > > When is / ever cross-compiled? > > If you think the answer is "never", then the entire column can be empty, and > you can omit it. That leaves you with a three column matrix to fill in. You would need two columns for /. One in case ROOT is native, one in case it is not. Hence it makes sense to write down two matrices, one for each usecase.
(In reply to comment #57) > (In reply to comment #53) > > For exheres-0, and probably the full-blown version of the DEPENDENCIES > > proposal > Is exheres-0 going into Gentoo? A quick googling brought me to Exherbo [1] > and it appears like exheres-0 has nothing to do with Gentoo anymore, but > with another distro, which is just an idea-fork of Gentoo and Portage? Hence > I think we do not need to talk about it here. Please correct me if I am > wrong. > > [1] http://exherbo.org/docs/exheres-for-smarties.html Some of the DEPENDENCIES proposals are heavily based upon exheres-0, and the exheres-0 dependencies format in turn started with some old work in Gentoo on solving the problem we're discussing. So you should look closely at this, since it's an existing and well-tested solution. > (In reply to comment #55) > So basically the same concept as libtool .la files or pkg-config's Libs — > indirect header dependencies and static libraries? See, we have to be careful with analogies, since they can easily lead you astray. It's best to think in terms of the big matrix. > You would need two columns for /. One in case ROOT is native, one in case it > is not. Hence it makes sense to write down two matrices, one for each > usecase. No, dependencies aren't expressed differently depending upon whether a package is or is not being cross-compiled. Thus you should write one matrix, and in the case where we're not concerned about cross-compiling, columns get merged (and in the case where we're not concerned about ROOT either, all the columns get merged).
I still don't see what't really wrong with current Gentoo dependencies plus HDEPEND,BADEPEND,IDEPEND. Lets consider some common types of dependencies: - Shared libraries go in RDEPEND and DEPEND. This includes most dlopen'd libraries because you most likely still need their headers during build, so you get struct definitions and other stuff. - Programs used at runtime only and not required for building go in RDEPEND (in rare cases PDEPEND to solve cycles). - Header-only libraries only (!) needed to build this package go to DEPEND. - If a package installs headers and the headers #include another package, it goes into BADEPEND (unless it's already in RDEPEND). - Tools used in pkg_{pre,post}inst and pkg_{pre,post}rm go to IDEPEND (and these are treated as / dependencies). (Have I missed anything?) I don't think a matrix can sufficiently represent dependencies. Those involve recursive logic not obvious from the field titles in the matrix, and the matrix also gives a false sense of complexity that doesn't really exist. Instead, I present some Prolog code which clearly defines the logic of this supposed dependency scheme: % Input predicates, which represent the state of the system at one point: % % installed(Package, Root) - package Package is physically present in root Root. % d_HDEPEND(Package, DependPackage) - Package has a HDEPEND on DependPackage % d_RDEPEND(Package, DependPackage) - ... has RDEPEND ... % d_DEPEND(Package, DependPackage) % d_PDEPEND(Package, DependPackage) % d_BADEPEND(Package, DependPackage) % d_IDEPEND(Package, DependPackage) % package Package is installed in root Root and has RDEPENDs satisfied installed_correctly(Package, Root) :- % must be physically installed installed(Package, Root), % all RDEPENDs must be installed correctly forall(d_RDEPEND(Package, RunDependPackage), installed_correctly(RunDependPackage, Root)). % package Package is installed correctly and all PDEPENDs are satisfied installed_completely(Package, Root) :- % must be physically installed installed(Package, Root), % all RDEPENDs must be installed completely forall(d_RDEPEND(Package, RunDependPackage), installed_completely(RunDependPackage, Root)), % all PDEPENDs must be installed completely forall(d_PDEPEND(Package, PostDependPackage), installed_completely(PostDependPackage, Root)). % package Package for root Root can be built right now can_be_built(Package, Root) :- % all HDEPENDs are satisfied forall(d_HDEPEND(Package, HostDependPackage), installed_correctly(HostDependPackage, '/')), % should this be installed_completely()? % all DEPENDs are satisfied forall(d_DEPEND(Package, DependPackage), can_be_built_against(DependPackage, Root)). % package Package in root Root can be built against, i.e. successfully used % within a DEPEND. can_be_built_against(Package, Root) :- % must be installed correctly installed_correctly(Package, Root), % all its BADEPENDs must be installed correctly forall(d_BADEPEND(Package, BuildAgainstPackage), installed_correctly(BuildAgainstPackage, Root)). % package Package has install dependencies satisfied for % installing or removing from root Root install_deps_satisfied(Package, Root) :- forall(d_IDEPEND(Package, InstallDependPackage), installed_correctly(InstallDependPackage, '/')). % should this be installed_completely()? This is not complete though; e.g. versioned dependencies and blockers are not considered. Still, you just *can't* call it complex.
> (Have I missed anything?) - Tools needed during compilation go in HDEPEND, obviously :)
... and BADEPEND is supposed to be recursive... fixed logic here http://ideone.com/bgPSH
(In reply to comment #59) > I still don't see what't really wrong with current Gentoo dependencies plus > HDEPEND,BADEPEND,IDEPEND. Lets consider some common types of dependencies: The primary concern is about what DEPEND means, and whether HDEPEND is a good idea, or whether TDEPEND is better. > I don't think a matrix can sufficiently represent dependencies. The matrix gives you the meaning of individual dependencies. Your recursive logic is covered by the term "usable" in PMS, and is deliberately kept separate, because it's fiddly. > and the matrix also gives a false sense of complexity that doesn't really exist. The matrix makes it easy for developers to understand what's going on. > (Have I missed anything?) You've missed every kind of library that is not a "shared object"... > Instead, I present some Prolog code which clearly defines the logic of this > supposed dependency scheme: Your notion of DEPEND is wrong at least in the "ROOT but not crosscompiling" case. This is one of the things we need to sort out, and the way to do it is to agree on what exactly each dependency kind means, and the way to do that is to fill in the matrix.
Putting the matrices aside, ferringb gave a nice explanation of how HDEPEND (or BUILDTOOL_/BDEPEND) should work: -- Preface -- He (and pkgcore) thinks in "domains". ROOT and the configuration used for it (i.e. contents of PORTAGE_CONFIGROOT) builds a domain. A domain may be foreign, i.e. cross-compiled, or native. A package manager working on some domain has a "master" PM, which may use another domain as a base. He may also be his own master. -- Behaviour of BDEPEND -- Now when it comes to dependency resolution, and a PM sees a BDEPEND, he will ask his master to install it into their ROOT. When it sees a DEPEND, it will simply install it into his own ROOT. If the PM is his own master, obviously BDEPEND will be installed into the same ROOT as DEPEND. ---- That is basically how it should work (apparently ferringb seconds this proposal), without needing to pull out The Big Matrix. IMO it could even be specified in this way in the PMS, since it defines the behaviour and is easy to understand.
Domains are simply the columns of the big matrix... Seriously, fill out the big matrix, and all shall become clear.
> The primary concern is about what DEPEND means, and whether HDEPEND is a > good idea, or whether TDEPEND is better. I think that we should first agree what *functionality* is right and then settle on the naming. We could argue to the end of time whether it's better to introduce HDEPEND or TDEPEND, and make DEPEND mean the other one. They do the same, just different names. I prefer HDEPEND because (as I've already mentioned) it would mean that packages with DEPEND==RDEPEND would remain correct under the new EAPI. > The matrix gives you the meaning of individual dependencies. Your recursive > logic is covered by the term "usable" in PMS, and is deliberately kept > separate, because it's fiddly. But when you're adding a feature and want it to work consistently on different PMs, precision is necessary. For example, BADEPEND needs to work recursively, and a new concept of "usable" needs to be added to make that work (in my code can_be_built_against). E.g. if P1 headers include P2 headers and P2 headers include P3 headers, then when you DEPEND (or TDEPEND) on P1, you want P2 and *also* P3. > You've missed every kind of library that is not a "shared object"... Please, provide some examples (and those that are not already a will-never-ever-cross-compile mess). > Your notion of DEPEND is wrong at least in the "ROOT but not crosscompiling" > case. This is one of the things we need to sort out, and the way to do it is > to agree on what exactly each dependency kind means, and the way to do that > is to fill in the matrix. Why the sudden need to support "native ROOT"? Who uses that? Gentoo Prefix maybe? I think for Prefix you could just modify the logic to make HDEPENDs and IDEPENDs require that packages be present in ROOT rather than in /. Then dependencies for Prefix would behave exactly as native, just everything would work within the prefix. So I've adjusted my Prolog specification to hopefully work with Prefix right: http://ideone.com/OCZ4h
(In reply to comment #65) > > The primary concern is about what DEPEND means, and whether HDEPEND is a > > good idea, or whether TDEPEND is better. > I think that we should first agree what *functionality* is right and then > settle on the naming. We could argue to the end of time whether it's better > to introduce HDEPEND or TDEPEND, and make DEPEND mean the other one. They do > the same, just different names. I prefer HDEPEND because (as I've already > mentioned) it would mean that packages with DEPEND==RDEPEND would remain > correct under the new EAPI. Sure. In that case, fill in the big matrix, ticking the boxes where you think there's a legitimate use case for that style of dependency. > But when you're adding a feature and want it to work consistently on > different PMs, precision is necessary. For example, BADEPEND needs to work > recursively, and a new concept of "usable" needs to be added to make that > work (in my code can_be_built_against). > E.g. if P1 headers include P2 headers and P2 headers include P3 headers, > then when you DEPEND (or TDEPEND) on P1, you want P2 and *also* P3. That comes down to getting the wording right in PMS. The thing is, that wording is only relevant to package mangler developers. We need something that ebuild developers can grasp. The big matrix, together with examples, is within the grasp of people who don't work on package mangler internals. Reading Prolog code is not. > > You've missed every kind of library that is not a "shared object"... > > Please, provide some examples (and those that are not already a > will-never-ever-cross-compile mess). Python modules. Ruby libraries. Haskell libraries. > > Your notion of DEPEND is wrong at least in the "ROOT but not crosscompiling" > > case. This is one of the things we need to sort out, and the way to do it is > > to agree on what exactly each dependency kind means, and the way to do that > > is to fill in the matrix. > > Why the sudden need to support "native ROOT"? Who uses that? Uh, that's been officially supported since before PMS existed.
Considering the amount of discussion done on the topic, I feel like it tries to go two mutually exclusive routes... So, what is our goal? Is it to express dependencies *exact* or *practically useful*. In other words, do we really want to express that large matrix, or just a minimal one necessary for our systems to work. For example, built-against dependencies are a neat thing for the former goal; but practically, I think they are a bit complex to implement. Considering that there are not many packages which will make a good use of them, do we really want this? Maybe it's acceptable to just use not-really-valid RDEPEND in that case.
We're not expecting every entry in the matrix to correspond to a dep. We're using the matrix to work out exactly what deps mean, and from that, how many different kinds of dependencies we need to make things work.
Ok, I've taken a stab at the Matrix. First I've just put down in plain text what I think stuff should do: DEPEND - When: when building this package - Where: wherever this package is HDEPEND: - When: when building this package - Where: in /, or wherever this package is for Prefix RDEPEND: - When: whenever this package is installed, during install, during uninstall - Where: wherever this package is BADEPEND: - When: when building a package that DEPENDs on this package, when building a package that DEPENDs on P1 which BADEPENDs on P2 ... which BADEPENDs on this package - Where: wherever this package is IDEPEND: - When: when installing or uninstalling this package, - Where: in /, or wherever this package is for Prefix PDEPEND: - When: eventually after this package is installed - Where: wherever this package is So if we assume that ROOT="wherever this package is", and that we're not on Prefix, the following matrix is obtained. Some details are lost however. If we're on Prefix, just move everything to "In ROOT". When it's needed Where it's needed On / In ROOT At fetch time At compile time HDEPEND DEPEND At install time IDEPEND RDEPEND At uninstall time IDEPEND RDEPEND When used as a compile dep BADEPEND Whenever installed RDEPEND At some point PDEPEND Changes to matrix rows: - I added "At uninstall time". - I removed "As a goal" and "When used as a dep", and instead added a single "Whenever installed". Again, if some of the matrix seems strange, remember again that "ROOT=wherever this package is", and re-read my original descriptions above, and maybe the Prolog logic. There's a question regarding the IDEPEND logic though. If we want to be able to reliably uninstall a package, its an IDEPEND should imply RDEPEND or HDEPEND or both, to make sure the IDEPEND'd package is already available. Else we might have to install new packages just to be able to uninstall a package (which is complex to implement, and can fail).
What I'm most interested now is if we can agree whether this is functionally sufficient, i.e. would you like it after arbitrary renaming. I'm also a little concerned about handling of RDEPENDs. I've always seen those as "needed whenever this is installed". Ciaran, why did you separate that into "When used as a dep" and "As a goal"? What *exactly* do these two mean?
Ok, second attempt, trying to fix RDEPEND to be less restrictive (my guess is that having it "whenever installed" causes problems with cycles). DEPEND - When: when building this package - Where: wherever this package is HDEPEND: - When: when building this package - Where: in /, or wherever this package is for Prefix RDEPEND: - When: - eventually after this package is installed ("as a goal") call these "When used as a dep": - when building a package which DEPENDs on this package - when building a package which DEPENDs on P1 which RDEPENDs on P2 ... which RDEPENDs on this package - when building a package which HDEPENDs on this package - when building a package which HDEPENDs on P1 which RDEPENDs on P2 ... which RDEPENDs on this package - Where: wherever this package is BADEPEND: - When: call these "When used as a compile dep": - when building a package that DEPENDs on this package, - when building a package that DEPENDs on P1 which BADEPENDs on P2 ... which BADEPENDs on this package - Where: wherever this package is IDEPEND: - When: when installing or uninstalling this package, - Where: in /, or wherever this package is for Prefix PDEPEND: - When: eventually after this package is installed - Where: wherever this package is I've tried to map the When's above to your wording Ciaran and got this: When it's needed Where it's needed On / In ROOT At fetch time At compile time HDEPEND DEPEND At install time IDEPEND At uninstall time IDEPEND When used as a compile dep BADEPEND When used as a dep RDEPEND As a goal RDEPEND At some point PDEPEND Again, for Prefix everything goes to In ROOT. I'm not sure if "ROOT!=/ but native" can be anything else than Prefix. Can it be? If so, what is the current semantic?
(In reply to comment #71) > Again, for Prefix everything goes to In ROOT. I'm not sure if "ROOT!=/ but > native" can be anything else than Prefix. Can it be? If so, what is the > current semantic? It's not Prefix. "ROOT!=/ but native" is how we use a stage3 to build a stage1 in an empty ROOT (see sys-apps/catalyst).
> It's not Prefix. "ROOT!=/ but native" is how we use a stage3 to build a > stage1 in an empty ROOT (see sys-apps/catalyst). Okay, I see that. I presume the non-prefix semantic I suggested can be made to work for that?
As far as producing stages goes, DEPEND currently means /, right? But my proposal is that DEPEND always means ROOT. If so, then when porting packages which are involved in building a stage1 to the new EAPI, some DEPENDs will have to be removed or changed to HDEPEND (based on use flags likely).
Yes, for stage builds, DEPEND is host build-time, RDEPEND is target build-time and run-time, and PDEPEND is target run-time-only (where host is ROOT=/ and target is ROOT!=/).
(In reply to comment #65) > > Your notion of DEPEND is wrong at least in the "ROOT but not crosscompiling" > > case. This is one of the things we need to sort out, and the way to do it is > > to agree on what exactly each dependency kind means, and the way to do that > > is to fill in the matrix. > Why the sudden need to support "native ROOT"? Who uses that? I do. It is useful if you work with containers, e.g. OpenVZ. (In reply to comment #66) > The big matrix, together with examples, is > within the grasp of people who don't work on package mangler internals. I disagree. > Reading Prolog code is not. I agree. See above for what ferringb had to say, and what I consider to be understandable. (In reply to comment #68) > We're not expecting every entry in the matrix to correspond to a dep. We're > using the matrix to work out exactly what deps mean, and from that, how many > different kinds of dependencies we need to make things work. s/We/I/ and you get it right. I still do not see what benefit The Big Matrix brings, and instead of filling out arbitrary forms, I would rather like to find any remaining flaws and then get this implemented, so we can all test it and see whether H/BDEPEND indeed provides working cross-compilation and prefix handling. Question to everyone: The summary mentions only HDEPEND. Why are you talking about BADEPEND and IDEPEND? I assumed this bug is about getting cross-compilation fixed, not about implementing every possible dependency style in the world…
> s/We/I/ and you get it right. I still do not see what benefit The Big Matrix > brings, and instead of filling out arbitrary forms, I would rather like to > find any remaining flaws and then get this implemented, so we can all test > it and see whether H/BDEPEND indeed provides working cross-compilation and > prefix handling. > > Question to everyone: The summary mentions only HDEPEND. Why are you talking > about BADEPEND and IDEPEND? I assumed this bug is about getting > cross-compilation fixed, not about implementing every possible dependency > style in the world… I second that! HDEPEND is a small change, orthogonal to BADEPEND and IDEPEND. The only reasons they popped up here seem to be that "it would confuse people to have so many *DEPENDs", and Ciaran mentioned something about "getting native ROOT right". I don't think any of those are preconditions to just getting HDEPEND. And it really wouldn't confuse people. A simple description like "used as build tool during compilation" would cover most cases.
(In reply to comment #66) > > > You've missed every kind of library that is not a "shared object"... > > > > Please, provide some examples (and those that are not already a > > will-never-ever-cross-compile mess). > > Python modules. Ruby libraries. Haskell libraries. They are DEPEND and RDEPEND, but not H/BDEPEND. Or is that wrong, too? (In reply to comment #71) > DEPEND > - When: when building this package > - Where: wherever this package is Into ROOT when ROOT is cross, and into / when it is not. ROOT can afaik not be expected to contain a compiler we could use to build packages, hence packages have to be build in /, hence that is where DEPEND needs to go. > HDEPEND: > - When: when building this package > - Where: in /, or wherever this package is for Prefix Into /. > RDEPEND: > - When: - eventually after this package is installed ("as a goal") Currently: Before that package is installed. I see no reason to change that. If you need to fix cycles, there is PDEPEND. > call these "When used as a dep": > - when building a package which DEPENDs on this package > - when building a package which DEPENDs on P1 which RDEPENDs on P2 > ... which RDEPENDs on this package > - when building a package which HDEPENDs on this package > - when building a package which HDEPENDs on P1 which RDEPENDs on P2 > ... which RDEPENDs on this package ??? If you care, please write me an email explaining this, to not crowd this report any more. > - Where: wherever this package is Into ROOT. To map it to ferringb's wording and adjust it to the nested-prefix case, just replace "/" with "master domain" and ROOT with "slave/target domain". Again, please correct me if there is something wrong with my perception of the needed behaviour (of H/BDEPEND and in relation to R-/DEPEND). (In reply to comment #77) > I second that! HDEPEND is a small change, orthogonal to BADEPEND and > IDEPEND. The only reasons they popped up here seem to be that "it would > confuse people to have so many *DEPENDs" So it appears I accidentally started this. Sorry!
> > DEPEND > > - When: when building this package > > - Where: wherever this package is > Into ROOT when ROOT is cross, and into / when it is not. > ROOT can afaik not be expected to contain a compiler we could use to build > packages, hence packages have to be build in /, hence that is where DEPEND > needs to go. Ok, but this "ROOT is not cross" behavior making DEPEND go to / would need to be triggered manually - not some arch guessing. I still want to be able to do a "cross build" x86->x86 exactly as if it was x86->arm. (ideally, I could bootstrap a clean stage1 (and stage3) entirely via cross-compilation, for the same architecture too) > > HDEPEND: > > - When: when building this package > > - Where: in /, or wherever this package is for Prefix > Into /. For Prefix, you just *can't* install into /, so you have to install to ROOT, for sure? Except when bootstrapping Prefix, which I would assume works similarly to normal bootstrapping, so the above mentioned trigger could be used to force to /. > > RDEPEND: > > - When: - eventually after this package is installed ("as a goal") > Currently: Before that package is installed. I see no reason to change that. Yes, that works too. > > call these "When used as a dep": > > - when building a package which DEPENDs on this package > > - when building a package which DEPENDs on P1 which RDEPENDs on P2 > > ... which RDEPENDs on this package > > - when building a package which HDEPENDs on this package > > - when building a package which HDEPENDs on P1 which RDEPENDs on P2 > > ... which RDEPENDs on this package > ??? I wasn't aware of the "before that package is installed" part above. If it was as I taught, these seemingly complex conditions would ensure a RDEPEND'd package is installed all right before we use it during compilation (just read it again, it's not written that bad :) . > > - Where: wherever this package is > Into ROOT. Yeah, but consider that ROOT temporarily changes to / while a host dependency is being installed. If such a package has a PDEPEND, you want it to go to / not to the cross root. I was just trying to be as precise as possible.
Created attachment 323962 [details, diff] HDEPEND for Portage 2.1.11.17 This is a port of HDEPEND to the latest Portage. The docs got lost in the process unfortunately for now. To use this in an ebuild, set EAPI=4-hdepend, otherwise this does nothing. Any eclasses used in such an ebuild may also need to be updated to handle this EAPI (good ones will complain about unknown EAPI). I've done a very simple test and it seems to work.
The "treat DEPEND as / dependencies even in EAPI=4-hdepend" switch is yet to be implemented though (the thing which bootstrapping will require).
(In reply to comment #81) > The "treat DEPEND as / dependencies even in EAPI=4-hdepend" switch is yet to > be implemented though (the thing which bootstrapping will require). bug #435066
Created attachment 323992 [details] emerge -pv dracut/asciidoc - logs (In reply to comment #80) > Created attachment 323962 [details, diff] [details, diff] > HDEPEND for Portage 2.1.11.17 This patch applies cleanly to sys-apps/portage-2.2.0_alpha128. I tried it with sys-kernel/dracut-023, which has DEPEND=asciidoc. I changed the ebuild to EAPI=4-hdepend and HDEPEND=asciidoc (did *not* remove asciidoc from DEPEND). Trying to emerge dracut like this appears like emerge --emptytree on the host (i.e. /) system — the output is quite different from a plain emerge -pv asciidoc there. Logs and infos attached.
Created attachment 323994 [details] emerge --info - host and target
> I tried it with sys-kernel/dracut-023, which has DEPEND=asciidoc. I changed > the ebuild to EAPI=4-hdepend and HDEPEND=asciidoc (did *not* remove asciidoc > from DEPEND). Trying to emerge dracut like this appears like emerge > --emptytree on the host (i.e. /) system — the output is quite different from > a plain emerge -pv asciidoc there. Logs and infos attached. This is to be expected. In EAPI=4-hdepend, DEPEND means install to target. You left asciidoc in DEPEND, so it tries to install asciidoc to target, along with a bunch of its dependencies. Just remove it from DEPEND and have it only in HDEPEND.
(In reply to comment #85) > This is to be expected. In EAPI=4-hdepend, DEPEND means install to target. > You left asciidoc in DEPEND, so it tries to install asciidoc to target, > along with a bunch of its dependencies. Just remove it from DEPEND and have > it only in HDEPEND. Isn't that a fairly strong argument in favour of TDEPEND and not HDEPEND?
(In reply to comment #85) > > I tried it with sys-kernel/dracut-023, which has DEPEND=asciidoc. I changed > > the ebuild to EAPI=4-hdepend and HDEPEND=asciidoc (did *not* remove asciidoc > > from DEPEND). Trying to emerge dracut like this appears like emerge > > --emptytree on the host (i.e. /) system — the output is quite different from > > a plain emerge -pv asciidoc there. Logs and infos attached. > > This is to be expected. In EAPI=4-hdepend, DEPEND means install to target. > You left asciidoc in DEPEND, so it tries to install asciidoc to target, > along with a bunch of its dependencies. Just remove it from DEPEND and have > it only in HDEPEND. I expected that asciidoc be installed into ROOT. See: [ebuild N ] app-text/asciidoc-8.6.5 to /usr/powerpc64-unknown-linux-gnu/ What I did not expect was portage to be installed to /: [ebuild N *] sys-apps/portage-2.2.0_alpha128 I mean, portage clearly is already installed into /, otherwise "emerge -pv asciidoc" above would not have worked…
> Isn't that a fairly strong argument in favour of TDEPEND and not HDEPEND? Lets take a look at dracut in particular: CDEPEND=">sys-fs/udev-166 dracut_modules_systemd? ( sys-apps/systemd ) " RDEPEND="${CDEPEND} ..." DEPEND="${CDEPEND} app-text/asciidoc >=dev-libs/libxslt-1.1.26 app-text/docbook-xml-dtd:4.5 >=app-text/docbook-xsl-stylesheets-1.75.2 virtual/pkgconfig " With HDEPEND, you get this: DEPEND=">sys-fs/udev-166 dracut_modules_systemd? ( sys-apps/systemd ) " RDEPEND="${DEPEND} ..." HDEPEND="app-text/asciidoc >=dev-libs/libxslt-1.1.26 app-text/docbook-xml-dtd:4.5 >=app-text/docbook-xsl-stylesheets-1.75.2 virtual/pkgconfig " With TDEPEND: TDEPEND=">sys-fs/udev-166 dracut_modules_systemd? ( sys-apps/systemd ) " RDEPEND="${TDEPEND} ..." DEPEND="app-text/asciidoc >=dev-libs/libxslt-1.1.26 app-text/docbook-xml-dtd:4.5 >=app-text/docbook-xsl-stylesheets-1.75.2 virtual/pkgconfig " I really don't see how one involves more work than the other, or any other significant difference.
(In reply to comment #88) > > Isn't that a fairly strong argument in favour of TDEPEND and not HDEPEND? > > Lets take a look at dracut in particular: You're not taking into account (DEPEND intersect RDEPEND).
> You're not taking into account (DEPEND intersect RDEPEND). In the original ebuild, (DEPEND intersect RDEPEND) is CDEPEND. In the modification this is renamed either to DEPEND or TDEPEND, which is the included in RDEPEND. Why do you think I'm not handling it? You mean if something from CDEPEND is actually a build tool, but I'm not putting it into HDEPEND? Doesn't look like it in this case. It would still be the same amount of work, TDEPEND or HDEPEND.
Created attachment 324000 [details] cross-emerge dracut resolution > What I did not expect was portage to be installed to / Yes, this is indeed wrong. I can't reproduce it though. I patched the dracut ebuild to use EAPI=4-hdepend and put asciidoc in both DEPEND and HDEPEND, but all looks nice (disregarding a block). Maybe this is a bug in portage 2.2? Please try 2.1.11.17.
Created attachment 324002 [details] dracut ebuild tested
I've played a little with this, and it seems one vital feature is missing: the ability to add "crosscompile only" dependencies. E.g. Python requires an existing Python interpreter, preferably the same version, to cross-compile. Tthe current solution which works badly is that the ebuild build a Python for the host and uses this. But this will fail as soon as you want to compile an external module. So Python would need something like: HDEPEND="crosscompile? ( ~${P}-${PV} )" You can't make it unconditional because it would produce a circular dependency when not cross compiling. The obvious solution would be to add a crosscompile USE flags, but the problem with this is that if you later boot up the system you've built and use emerge inside, "emerge --newuse" will want to recompile everything which has the crosscompile flag. Is there any support for conditional dependencies which are not use flags? I'm thinking something like "emerge --crosscompile ..." which would enable this dependency but not actually save a USE flag into the built package.
Created attachment 324038 [details, diff] HDEPEND for Portage 2.1.11.17, now with --crosscompile and CROSS_HDEPEND This updated HDEPEND patch introduces a --crosscompile flag, with the following semantic: For old EAPI, behavior is unchanged, i.e. DEPEND defaults to / subject to --root-deps. For new EAPI, without --crosscompile: - DEPEND means /. - CROSS_HDEPEND dependencies are ignored. For new EAPI, with --crosscompile: - DEPEND means ROOT. - CROSS_HDEPEND dependencies are in effect, equivalent to HDEPEND This behavior means that Catalyst and other "native ROOT" cases will continue working without having to pass any new arguments to emerge. For cross building however, a --crosscompile flag to emerge is necessary which adjusts the semantic of DEPEND. Old-EAPI cross-compilable packages will remain cross-compilable, because DEPEND=/ will be preserved, and new EAPI packages will use DEPEND=ROOT. The --crosscompile flag also enables CROSS_HDEPEND, which are host dependencies (equivalent to HDEPEND), but only when cross compiling. I've added this because many packages need themselves present on the host for cross-compiling, but this can't work for native builds.
(In reply to comment #93) > HDEPEND="crosscompile? ( ~${P}-${PV} )" Recently on IRC, ferringb proposed exactly that - a crosscompile flag (I think part of his proposal with foo@bar? more-than-use-flags). I am not sure about the details though. Could look it up in the history, if necessary.
(In reply to comment #95) > (In reply to comment #93) > > HDEPEND="crosscompile? ( ~${P}-${PV} )" > Recently on IRC, ferringb proposed exactly that - a crosscompile flag (I > think part of his proposal with foo@bar? more-than-use-flags). I am not sure > about the details though. Could look it up in the history, if necessary. Yes, that may be a good idea. I don't like adding more *DEPEND variables too. But CROSS_DEPEND was the easiest to implement quickly. But still, this latest patch should work well and not break anything.
I've set up an overlay where I keep patches related to cross compilation: http://code.google.com/p/ambro-cross-overlay/ On this page I put some instructions for cross-compiling a complete @system for ARMv6. It should work without any errors. The overlay contains (currently): - Portage patched with my second patch (portage-2.1.11.17-hdepend-2.patch). - Crossdev, added some ac_ variables for Python, and my cross-emerge-ng emerge wrapper. This one basically just wraps compilers with --sysroot option and calls emerge right. - Patched packages to cross-compile (groff, pam, python). PAM and Python ebuilds use EAPI=4-hdepend to get dependencies right.
(In reply to comment #94) > For cross building however, a --crosscompile flag to emerge is necessary > which adjusts the semantic of DEPEND. Old-EAPI cross-compilable packages > will remain cross-compilable, because DEPEND=/ will be preserved, and new > EAPI packages will use DEPEND=ROOT. The --root-deps option varies DEPEND semantics because it tweaks the behavior of *existing* EAPIs. When you add a *new* EAPI (4-hdepend in this case), you shouldn't have to resort to such hacks. It's just wrong.
> The --root-deps option varies DEPEND semantics because it tweaks the > behavior of *existing* EAPIs. When you add a *new* EAPI (4-hdepend in this > case), you shouldn't have to resort to such hacks. It's just wrong. Ok, I was just trying to be non-invasive. I taught Catalyst and other "native ROOT" setups required this behavior (to have / satisfy some dependencies), but after looking at glibc and gcc, it seems just moving some stuff out of DEPEND into HDEPEND is sufficient for bootstrapping, with DEPEND always meaning ROOT. But then I've heard people use "native ROOT" setups for other purposes than bootstrapping (containers?). I'm afraid these really would break if DEPEND always meant ROOT, assuming the dependencies were not made conditional on that specific use case. Can someone who knows about this explain the situation? I really don't see the --crosscompile option itself as a hack. You're just telling emerge what you're doing. Since, really, emerge can't reliably guess whether you want to cross-compile or not.
So basically, I'd like to know if it's OK for DEPEND to *always* mean ROOT in new EAPI, including unusual setups that must continue to work. If so, the --crosscompile would then only enable CROSS_HDEPEND dependencies and not mess with the meaning of DEPEND.
(In reply to comment #99) > But then I've heard people use "native ROOT" setups for other purposes than > bootstrapping (containers?). I'm afraid these really would break if DEPEND > always meant ROOT, assuming the dependencies were not made conditional on > that specific use case. Can someone who knows about this explain the > situation? container-ROOT is basically the same as bootstrapping, I think. You usualy do not have a cross-compiler installed into ROOT, and hence cannot assume the used (i.e. /) compiler to find headers and libraries inside ROOT. As long as there is no reliable way of telling him (and pkg-config, etc) at runtime to search for stuff inside ROOT instead /, DEPEND would have to go into /. This would of course only work if CHOST in / and ROOT match and both have CHOST=CTARGET, i.e. no cross-compilation. Since Ambroz mentioned the case where ROOT has a matching CHOST and still uses a cross-compiler: You could decide the location for DEPEND automatically if there is no crosscompiler installed to ROOT - which should be easily detectable by the presence of cross-*/* packages inside ROOT. (Provided the condition on CHOST/CTARGET is met.) If the compiler (at least gcc, clang, icc) and the rest of the toolchain (pkg-config, cmake?, ...) can be reliably instructed to look for their stuff inside ROOT, the autodetection would not be necessary and DEPEND could always go into ROOT.
> You usualy do not have a cross-compiler installed into ROOT, and hence > cannot assume the used (i.e. /) compiler to find headers and libraries > inside ROOT. As long as there is no reliable way of telling him (and > pkg-config, etc) at runtime to search for stuff inside ROOT instead /, > DEPEND would have to go into /. You *don't* need to have a cross-compiler installed to ROOT to cross-compile completely within ROOT. All you need is add a --sysroot compile flag to gcc/g++, and your cross-compiler will forget about /usr/CHOST/lib and /usr/CHOST/include. Actually, this is exactly what I do in the instructions I've posted. And it works great - you can cross-compile a complete @system into an empty directory, not dirtied by libraries from your cross compiler. This means you cross-emerge a glibc and gcc into ROOT and they will be package-managable (but you have to be careful they're the same version as your cross compiler). > Since Ambroz mentioned the case where ROOT has a matching CHOST and still > uses a cross-compiler: > You could decide the location for DEPEND automatically if there is no > crosscompiler installed to ROOT - which should be easily detectable by the > presence of cross-*/* packages inside ROOT. (Provided the condition on > CHOST/CTARGET is met.) As said, not reliable. I would consider this magic guessing a hack, and --crosscompile a reliable solution. > If the compiler (at least gcc, clang, icc) and the rest of the toolchain > (pkg-config, cmake?, ...) can be reliably instructed to look for their stuff > inside ROOT, the autodetection would not be necessary and DEPEND could > always go into ROOT. - gcc: yes, see --sysroot above - clang, icc: I'm guessing no (but do we even care for such fringe cases?) - pkg-config: yes, this works automagically via CHOST-pkg-config which wraps pkg-config to work relative to SYSROOT. The wrapper is created by crossdev when you build the cross-toolchain. - cmake: has cross-compile support (google "cmake cross compiling"), and AFAIK gentoo makes proper use of it
> - gcc: yes, see --sysroot above I think that also needs glibc installed into ROOT, else the linker won't be able to link anything. You can install glibc itself without --sysroot gcc option, and from this point use --sysroot for everything. (see my instructions. They really work!)
Ok, so container-ROOT needs no special DEPEND location handling. I will try your --sysroot approach there. I will also try the --sysroot approach in the cross-compile setting, and report back. Formerly setting SYSROOT failed here, but maybe with HDEPEND and --sysroot it starts to work.
Ok, just be sure to follow the instructions exactly. If you do that, it should work completely without errors :)
(In reply to comment #105) > Ok, just be sure to follow the instructions exactly. If you do that, it > should work completely without errors :) P.S: Afterwards crossdev should be updated.
(In reply to comment #99) > I really don't see the --crosscompile option itself as a hack. You're just > telling emerge what you're doing. Since, really, emerge can't reliably guess > whether you want to cross-compile or not. The --crosscompile option it not the problem. The problem is the fact that it changes the ROOT where DEPEND is installed. As I tried to explain, changing the ROOT based on options is a hack, and should not be necessary in cross-aware EAPI. In the cross-aware EAPI, you should be using two separate variables instead of modifying the behavior of one variable.
For example, you could split out a CROSS_TDEPEND variable. That way, the ebuild can set CROSS_TDEPEND=${DEPEND} if appropriate, or it can have different dependencies in those two variables.
Attachment #324038 [details, diff] produces a broken portage on my system: Traceback (most recent call last): File "/usr/bin/emerge", line 47, in <module> retval = emerge_main() File "/usr/lib64/portage/pym/_emerge/main.py", line 2071, in emerge_main myopts, myaction, myfiles, spinner) File "/usr/lib64/portage/pym/_emerge/actions.py", line 302, in action_build settings, trees, myopts, myparams, myaction, myfiles, spinner) File "/usr/lib64/portage/pym/_emerge/depgraph.py", line 7364, in backtrack_depgraph myaction, myfiles, spinner) File "/usr/lib64/portage/pym/_emerge/depgraph.py", line 7396, in _backtrack_depgraph success, favorites = mydepgraph.select_files(myfiles) File "/usr/lib64/portage/pym/_emerge/depgraph.py", line 2280, in select_files self._load_vdb() File "/usr/lib64/portage/pym/_emerge/depgraph.py", line 565, in _load_vdb vardb.aux_get(pkg.cpv, []) File "/usr/lib64/portage/pym/_emerge/FakeVartree.py", line 127, in _aux_get_wrapper self._portdb.aux_get(pkg, self._portdb_keys, myrepo=repo))) File "/usr/lib64/portage/pym/portage/dbapi/porttree.py", line 442, in aux_get mydata, ebuild_hash = self._pull_valid_cache(mycpv, myebuild, mylocation) File "/usr/lib64/portage/pym/portage/dbapi/porttree.py", line 377, in _pull_valid_cache metadata = auxdb[cpv] File "/usr/lib64/portage/pym/portage/cache/template.py", line 50, in __getitem__ self.validation_chf, paths=self.store_eclass_paths) File "/usr/lib64/portage/pym/portage/cache/template.py", line 274, in reconstruct_eclasses eclasses = eclass_string.rstrip().lstrip().split("\t") AttributeError: 'NoneType' object has no attribute 'rstrip' Attachment #323962 [details, diff], however, works. Version is 2.2.0_alpha128.
(In reply to comment #83) > I tried it with sys-kernel/dracut-023, which has DEPEND=asciidoc. I changed > the ebuild to EAPI=4-hdepend and HDEPEND=asciidoc (did *not* remove asciidoc > from DEPEND). Trying to emerge dracut like this appears like emerge > --emptytree on the host (i.e. /) system — the output is quite different from > a plain emerge -pv asciidoc there. Logs and infos attached. Ambroz figured this out: if you pass --root-deps=rdeps, this means that DEPEND is ignored. But it also makes emerge not load the VDB on the /
Created attachment 324142 [details, diff] HDEPEND for Portage 2.1.11.17 This patch has the following changes from the previous one: - DEPEND for EAPI=4-hdepend now always goes into ROOT, no matter whether --crosscompile is used. - Fixed bug from comment 83, 87. The problem was an optimization in emerge which avoided loading the VDB from / if --root-deps=rdeps was given. Previously, this was OK, because --root-deps=rdeps implied that there would be no / dependencies, but now HDEPEND also expresses / dependencies which cannot be disabled by --root-deps=rdeps. - Added --host-deps=<y/n> option. With --host-deps=n, all / dependencies are automatically ignored. This includes HDEPEND dependencies and, in old EAPI, DEPEND dependencies when --root-deps is not present. This option might be useful for example when bootstrapping Gentoo or Gentoo Prefix on systems which are not Gentoo. (NOTE: when I say /, I mean, opposed to ROOT. Even if these two happen to be the same, the distinction still exists. E.g. --host-deps=n doesn't disable all dependencies on native.) The patched ebuilds for portage are available in an overlay at http://code.google.com/p/ambro-cross-overlay/ . There are also some updated ebuilds there which utilize this new dependency support.
Concerning the error in comment 109, we have not been able to reproduce it on any other system. If anyone else is experiencing this, please give as much info as possible about your system.
The --host-deps option looks questionable. Why should the user be able to disable installation of host dependencies? Wouldn't that trigger a build failure if the dependencies are unsatisfied? If so, isn't it better to detect the unsatisfied host dependencies during dependency resolution and bail out then if necessary?
(In reply to comment #113) > The --host-deps option looks questionable. Why should the user be able to > disable installation of host dependencies? Wouldn't that trigger a build > failure if the dependencies are unsatisfied? If so, isn't it better to > detect the unsatisfied host dependencies during dependency resolution and > bail out then if necessary? You mean let the user craft a /etc/portage/profile/package.provided file, if the host distro is no Gentoo? Sounds a bit cumbersome, but still like a good way in between (in between making portage install stuff to non-Gentoo and running into build failures because of unchecked deps).
If they're using another distro then they can use a prefix or a chroot (like for chromium-os, they use a Gentoo chroot to build on Ubuntu).
Zac, consider they're bootstrapping Prefix on a weird system for which Gentoo doesn't currently exist, in any form.
Note that --host-deps is not a new feature. It has always been possible to disable all host dependencies via --root-deps=rdeps - my intent was to keep this possible. But because --root-deps is tied to the lack of distinction of host/target deps, and will (hopefully) be removed someday, I added --host-deps on top. Do note that --host-deps=n implies the behavior of --root-deps=rdeps, but disables HDEPEND for new EAPI as well.
> note that --host-deps=n implies the behavior of --root-deps=rdeps Sorry, that's a little imprecise. What I meant to say is: if --root-deps is not present (DEPEND means / by default), --host-deps=n disables DEPEND (since in that case DEPEND is treated as host deps and user wants to disable host deps).
Okay, I suppose that the --host-deps option is acceptable, but the documentation will need to have a big warning to discourage people from using it (and suggest alternatives like package.provided). Also, when it is committed to portage, we'll have to separate it into different commit from the HDEPEND/EAPI patch, since it's a feature that stands by itself. Since your patch hasn't excluded EAPI 4-hdepend from any of the conditionals that enable EAPI 5 features, it would make sense to rename 4-hdepend to 5-hdepend. Alternatively, you add EAPI 4-hdepend to the relevant conditionals to disable EAPI 5 features. You can locate these conditionals by searching for 4-python and 4-slot-abi in the sources.
For the HDEPEND EAPI, we might also consider adjusting the definition of RDEPEND, so that it's strictly required at build-time. That way target build+run deps can simply be listed in RDEPEND, instead of having to list them in both DEPEND and RDEPEND. Then, for pure run-time-only target deps, PDEPEND should be used instead of RDEPEND.
I think that might interfere with installation of build tools needed to build a package. Consider: - Package A needs build tool B (HDEPEND). - B needs runtime-only tool C (PDEPEND, per your suggestion). It is guaranteed that during the build of A, built tool B will be installed. But, will *C*, which B requires to work, be installed? What is the current behavior regarding this? Do PDEPEND dependencies have to be satisfied for packages in DEPEND?
(In reply to comment #121) > What is the current behavior regarding this? Do PDEPEND dependencies have to > be satisfied for packages in DEPEND? Yes, they they have to be. Do it should work just fine. The new RDEPEND behavior can be easily implemented in _add_pkg_deps by simply adding RDEPEND twice: once with runtime=True and once with buildtime=True.
(In reply to comment #119) > Since your patch hasn't excluded EAPI 4-hdepend from any of the conditionals > that enable EAPI 5 features, it would make sense to rename 4-hdepend to > 5-hdepend. Alternatively, you add EAPI 4-hdepend to the relevant > conditionals to disable EAPI 5 features. You can locate these conditionals > by searching for 4-python and 4-slot-abi in the sources. While at it, since we're not diverging from the original EAPI but adding features to it, could we please name it N+hdepend?
We've got support for experimental EAPI 5-hdepend in portage-2.1.11.22 and 2.2.0_alpha133: http://blogs.gentoo.org/zmedico/2012/09/25/experimental-eapi-5-hdepend/
I really like this work. I'm concerned, however, about the name scheme chosen, and the change in meaning of current usage. ATM, DEPEND = build-machine dependency. This will be the "new" HDEPEND. CHOST DEPEND (ie must be installed in runtime ROOT during build, a new category) is what DEPEND will be under the proposed new EAPI. It seems like you're changing fundamentally what DEPEND means, when it would appear to make more sense for the new variable HDEPEND to mean the new category of packages. This is equivalent to using TDEPEND, but would be more consistent with CBUILD and CHOST which are the relevant variables for everything but the toolchain. To show that thinking only in terms of host and target is not useful in this context, consider that cross-compile for anything but toolchain is: CBUILD != CHOST, or equivalently CBUILD != CTARGET *not* CHOST != CTARGET (which only ever applies to a Canadian-cross toolchain.) It'd be better if we were consistent throughout the software, and thus also consistent with upstream packages. The change in what DEPEND means is however a much larger concern, since it means that ebuild authors will have to keep two completely different (almost opposite) meanings for it in mind, depending on EAPI. That sounds like a maintenance nightmare waiting to happen. Like I said, though, I like the underlying work a lot. Kudos.
(In reply to comment #125) > It seems like you're changing fundamentally what DEPEND means, when it would > appear to make more sense for the new variable HDEPEND to mean the new > category of packages. Practical tests in the ambro-cross overlay have shown, that very little changes to ebuilds are necessary if DEPEND means target dependencies. It appears that most packages need the DEPEND dependencies installed inside ROOT to build, not on the host. The possible explanation for this is that DEPEND most often contains libraries and only very seldomly tools that need to be executed during build. (Please read about the details above.)
(In reply to comment #126) P.S: The idea of adhering to the CBUILD/CHOST/CTARGET naming scheme was abandoned, because it appears this scheme is only used by autotools, not the most intuitive and also complicating things for the common case (non-canadian cross). While we *could* destinguish between "build" and "target" to be closer to autotools naming (with HDEPEND becoming BDEPEND for Build-host DEPENDencies), this option was never really considered. From counting the number of suggestions and the number of supporters for each, no strong majority could be determined. So it is assumed that the actual naming is purely a matter of opinion and there is no technically superior solution.
> Practical tests in the ambro-cross overlay have shown, that very little > changes to ebuilds are necessary if DEPEND means target dependencies. It > appears that most packages need the DEPEND dependencies installed inside ROOT > to build, not on the host. The concern is not about whether ebuilds can be written in the new style, nor whether they can easily be transformed. The problem is that the same variable means completely different things depending on the EAPI. And this is not some new setting that's only been around for a while: it's been there since the beginning, and is one of the fundamental variables. That semantic dissonance cannot be addressed unless you leave DEPEND meaning what it currently does, and use another variable for the new category of dependency. > The possible explanation for this is that DEPEND most often contains libraries > and only very seldomly tools that need to be executed during build. (Please > read about the details above.) I have read this bug a few times, and the linked ones ;) Ambro in fact raised that some current DEPENDs (presumably those not also listed in RDEPEND) would need to be changed, and that the meaning of the variable is different according to EAPI. He was originally in favour of TDEPEND, but ease of conversion swung it (for him). What you state here is more of a case for using some sort of LIB_DEPEND which would imply sub-slot := by default, and the requirement that it be installed in ROOT, before build, as well as being required at runtime. Leaving that aside, the new category, which must be installed in target root before build, can be called what you like: TDEPEND is fine too. But the semantic problem remains. "HDEPEND becoming BDEPEND for Build-host DEPENDencies" is just silly: DEPEND already has exactly that meaning. It's just unfortunate that LIB_DEPEND was not imported at the same as DEPEND was (from BUILD_DEPEND), in the original ports conversion, since it meant DEPEND is used to specify everything "required at build-time" whereas libraries need special treatment (like being installed in the target ROOT.) > The idea of adhering to the CBUILD/CHOST/CTARGET naming scheme was abandoned, > because it appears this scheme is only used by autotools Well that means it is used by gcc and binutils, as well as most portable software, including all GNU userland tools, which form the basis of Gentoo. It is also used within Gentoo and portage itself, as a result (see toolchain*.eclass for example.) I accept however that embedded people think in terms of (build)host and target, and it doesn't matter to me if you wanted to call it a target depend. But please, don't change the fundamental meaning of DEPEND ten years into the format. (In reply to comment #120) > For the HDEPEND EAPI, we might also consider adjusting the definition of > RDEPEND, so that it's strictly required at build-time. That way target > build+run deps can simply be listed in RDEPEND, instead of having to list them > in both DEPEND and RDEPEND. Isn't that what the proposed new "DEPEND" (or TDEPEND in current terms) does as well?
(In reply to comment #128) > I accept however that embedded people think in terms of (build)host and > target, > and it doesn't matter to me if you wanted to call it a target depend. But > please, don't change the fundamental meaning of DEPEND ten years into the > format. There's no difference other than for cross-compilation, and cross-compilation has been poorly supported for as long as DEPEND has existing (see the emerge --root-deps option, which can be used to tweak DEPEND behavior for older EAPIs).
(In reply to comment #127) the concept of build/host/target isn't autotool specific, it just popularized and standardized it. non-autotool packages that support cross-compiling have "build" tools (same as autotool's concept of build), and then generally merge the concept of "host" and "target" since very few packages will actually differ there. compilers tend to be the only ones where "host" and "target" are different. further, the autotool definition directly corresponds to the concepts of handling of targets in the PM itself. this is why we have CBUILD/CHOST/CTARGET and why ebuilds/eclasses use those variables in selecting the correct tool. as for one being superior to the other (BDEPEND vs HDEPEND), i'd say being self-consistent is a fairly strong marker. it leads to pointless confusion and inconsistent implementations. looking back to the original example with libxcb: CBUILD=x86_64-linux-gnu CHOST=arm-linux-gnu doxygen: needed in ROOT=/ to be used by $CBUILD code xcb-proto: needed in ROOT=/usr/$CHOST so we can compile against it libXau: needed in ROOT=/usr/$CHOST so we can link & run against it so with a BDEPEND name, the natural split would be: BDEPEND=doxygen DEPEND='xcb-proto libXau' RDEPEND=libXau while with a HDEPEND name, the split would be: DEPEND=doxygen HDEPEND='xcb-proto libXau' RDEPEND=libXau what would be wrong would be to use: HDEPEND=doxygen DEPEND='xcb-proto libXau' RDEPEND=libXau
Turns out that HDEPEND/BDEPEND support is actually needed to let emerge properly detect dependency cycles. For example, consider a cycle (in the currently implemented stable EAPI): pkg1: RDEPEND="pkg2" DEPEND="${RDEPEND}" pkg2: RDEPEND="pkg1" DEPEND="${RDEPEND}" It was intended that when pkg1 is build, pkg2 should be present in ROOT at build- and run-time, e.g. it is a shared library. If there was no cycle, e.g.. pkg2 didn't depend back on pkg1, emerge would *probably* install pkg2 before pkg1. However in case of a cycle, or for some other obscure reason, emerge may decide that it won't satisfy the RDEPEND pkg1-->pkg2 at build time, since it is, by definition, a run-time dependency. On the other hand, the build-time dependency DEPEND pkg1-->pkg2 will be satisfied by pkg2 on the build system. To sum up, emerge can, and sometimes does, end up building pkg1 while pkg2 is not installed in ROOT, which fails. (note this only applies to running emerge without any --root-deps option) The fix is to port the ebuilds to HDEPEND/BDEPEND. E.g. with HDEPEND as it is currently implemented, or if it was renamed to BDEPEND (first or third option in comment 130): pkg1: RDEPEND="pkg2" DEPEND="${RDEPEND}" (DEPEND means build-time ROOT dependency) pkg2: RDEPEND="pkg1" DEPEND="${RDEPEND}" Or, with HDEPEND with autotools semantic (second option in commend 130): pkg1: RDEPEND="pkg2" HDEPEND="${RDEPEND}" (HDEPEND means built-time ROOT dependency) pkg2: RDEPEND="pkg1" HDEPEND="${RDEPEND}" Either way, there will be a strict build-time ROOT dependency cycle, and emerge will fail (as opposed to a RDEPEND-only dependency cycle, which will not be an error because emerge is allowed to not satisfy RDEPEND's on build time, even though it will make an effort to).
(In reply to comment #129) >> please, don't change the fundamental meaning of DEPEND ten years into the >> format. > There's no difference other than for cross-compilation, and cross-compilation > has been poorly supported for as long as DEPEND has existed Yes, a consequence of the original design decision to conflate BUILD_DEPEND with LIB_DEPEND for ease of implementation, since cross-compiling and ABI changes were clearly not a concern of whoever decided to merge them into DEPEND. However, we are where we are. > (see the emerge --root-deps option, which can be used to tweak DEPEND behavior > for older EAPIs). As you said yourself at least a couple of times, that's an awful hack, or at very least a sledgehammer. Can we at least agree that by default, DEPEND means "build-time dependency installed to /"? After all, as you said in comment #75: > for stage builds, DEPEND is host build-time, RDEPEND is target build-time and > run-time, and PDEPEND is target run-time-only (where host is ROOT=/ and target > is ROOT!=/) So it's clear that the notion of DEPEND = build-host ROOT = / (ie: CBUILD) permeates Gentoo tools, as well as ebuilds and extant EAPIs, and thus developer thinking. Now with DEPEND = CHOST, ROOT = /path/to/target (however you do it, tho as SpanKy pointed out the current naming is the one wrong choice) that changes radically. So we don't get self-consistency across the whole domain. Over time, that leads to problems, especially for people who don't know the history. Ideally they'd have read this bug and others, and understood the issues, specifically around the difference between CHOST and CTARGET, and why neither is CBUILD; and why it was decided to make DEPEND mean different things according to EAPI (which baffles me from a technical pov.) In fact, if you changed the title of this bug to: "[Future EAPI] New variable for classifying build-time dependencies as CHOST vs CBUILD" ..it would be much clearer what is being discussed, within the context of ebuilds, portage, and make.conf. It would be much more technically accurate as well; CBUILD is the current default. I don't buy the conversion argument either, having thought about it. Regardless of how little changes, when you convert from the existing setup to support cross-compiling, you MUST review the ebuild and decide which category the dependency belongs in. In that case, it is just as easy to change DEPEND= to HDEPEND= and move any CBUILD ones to a new DEPEND. The time will be spent on reviewing and deciding where they belong, not in the variable conversion, which is _trivial_. (They're editing the ebuild already to change EAPI; if they don't want to review the dependencies, they have bigger problems.) Ebuilds with current EAPI-5-hdepend are even easier to convert and that can be done automatically by downstream, who have resources to put into it, tho it's a trivial sed script. No doubt they'd want to test it before deployment. Anyhow, would be nice to get this moving, and have support for this, that is technically consistent with the rest of portage and the distribution. After all the implementation work has already been done.
Idea introduced by ulm on IRC: There are three variables: DEPEND, HDEPEND, TDEPEND. DEPEND and the HDEPEND/TDEPEND pair are mutually exclusive (if DEPEND is set, neither of the others may be, and if HDEPEND or TDEPEND is set, DEPEND may not be). Thus current ebuilds can continue to use the simple DEPEND, but if a maintainer wants to support cross compilation, he can simply split DEPEND into T- and H-. Seems like the most simple conversion path.
Please don't make this any more complex than it needs to be. This reminds me of the ol' RDEPEND=${DEPEND} automagic (or was it other way around?). Let's just add HDEPEND, and people who are working on cross-compilation will shuffle dependencies as necessary when bumping EAPI.
(In reply to Dennis Schridde from comment #133) Nice that I'm credited for this, but on IRC I've only said that we could either go for DEPEND/HDEPEND or for TDEPEND/HDEPEND. This was mentioned above in comment #0, so nothing new.
(In reply to Michał Górny from comment #134) > Please don't make this any more complex than it needs to be. Agreed; it's as complex in portage as it has to be, and no more: CBUILD, CHOST and CTARGET exactly as in autoconf and the GNU toolchain. > Let's just add HDEPEND, and people who are working on cross-compilation will > shuffle dependencies as necessary when bumping EAPI. Please clarify which option you are referring to. If you mean HDEPEND = CHOST dependencies then I concur. If you mean the existing prototype "HDEPEND" implementation, then I most emphatically do not.
I don't really care which deps are put where as long as simple developers can bump EAPI without needing to learn the details of cross-compilation. Cross-compilation is an optional feature and dedicated team of people should play with it. That said, RDEPEND+DEPEND needs to keep working for a regular, non-cross system. DEPEND may mean either CBUILD or CHOST deps, this is not really important to me. To keep it simple, one additional variable that cross folks could use to split proper deps out of DEPEND. Nothing more than that.
(In reply to Michał Górny from comment #137) > That said, RDEPEND+DEPEND needs to keep working for a regular, non-cross > system. DEPEND may mean either CBUILD or CHOST deps, this is not really > important to me. To keep it simple, one additional variable that cross folks > could use to split proper deps out of DEPEND. Nothing more than that. That would be HDEPEND = CHOST dependencies; the other option requires a change of meaning in DEPEND depending on which EAPI you are using. And as you said, cross-compile folks can shuffle dependencies (or in this case, chromium folks can, since conversion from the other format is trivial.)
(In reply to Steve L from comment #138) > (In reply to Michał Górny from comment #137) > > That said, RDEPEND+DEPEND needs to keep working for a regular, non-cross > > system. DEPEND may mean either CBUILD or CHOST deps, this is not really > > important to me. To keep it simple, one additional variable that cross folks > > could use to split proper deps out of DEPEND. Nothing more than that. > > That would be HDEPEND = CHOST dependencies; the other option requires a > change of meaning in DEPEND depending on which EAPI you are using. > > And as you said, cross-compile folks can shuffle dependencies (or in this > case, chromium folks can, since conversion from the other format is trivial.) Just an idea, but would it be possible to have portage calculate DEPEND as HDEPEND + DEPEND if not cross-compiling? That way, people can keep just using RDEPEND + DEPEND, and DEPEND can then later get split by people having the need for cross-compiling, without it affecting anyone else.
> Just an idea, but would it be possible to have portage calculate DEPEND as > HDEPEND + DEPEND if not cross-compiling? Why? If someone adds HDEPEND to an ebuild in order to fix cross-compiling, it should not break anything for those who aren't. The only difference is that you may end up with more packages emerged to ROOT but those packages should be there anyway. Even if you aren't cross-compiling, you should build against headers in ROOT and not in / because the configuration may be different. You should only rely on / to run binaries at build time and that already applies whether you are cross-compiling or not - ${ROOT}/usr/bin is not in the PATH.
Actually I take that back slightly. You could say that the configuration would only be different if PORTAGE_CONFIGROOT != /. So many different ways of looking at this. :)
(In reply to Bjarke Istrup Pedersen from comment #139) > (In reply to Steve L from comment #138) > > That would be HDEPEND = CHOST dependencies; the other option requires a > > change of meaning in DEPEND depending on which EAPI you are using. > > Just an idea, but would it be possible to have portage calculate DEPEND as > HDEPEND + DEPEND if not cross-compiling? Well that is in fact exactly the result you get (it couldn't work any other way, I mean.) If you're building for the native machine, CHOST = CBUILD, so if it's a HDEPEND which needs to be installed before the package can build, and ROOT=/ then the only place it can be installed is the native machine, which means it's effectively a DEPEND. (We build software to run on a host somewhere; in this case the same host we're building on.) This reasoning lies behind the thinking which merged LIB_DEPEND into BUILD_DEPEND and called it DEPEND in portage, without realising that you don't always want them on the native machine, and in fact it's essential to track libs separately (which led to the "thinking" that gave us slot operators instead of realising there was a massive hole in the spec.) It's not precisely the same thing, in that as James said, the configuration for the "to be installed" set is not the same as "what is currently installed", which matters if you're building in a ROOT, which may well be the current machine, eg if you're testing an upgrade. But then having HDEPEND (+ LIB_DEPEND if you want automatic slot dependencies, or more broadly better lib-handling) in addition is precisely what you need, same as if you were cross-compiling into a ROOT. Either way a library dependency (or a header dep which must be installed in the ROOT prior to build) is simply not the same as a build dependency like make or autoconf. > That way, people can keep just using RDEPEND + DEPEND, and DEPEND can then > later get split by people having the need for cross-compiling, without it > affecting anyone else. Indeed. Ebuilds can be made cross-compilable one at a time, or in groups: the whole tree doesn't need to change at once, nor would it help anything, since we want some QA and testing before it goes in. @James: http://forums.gentoo.org/viewtopic-t-986644.html (figured I was getting a bit o-t for the bug itself, and better if we discuss those aspects there.)
(In reply to Dennis Schridde from comment #133) > Idea introduced by ulm on IRC: There are three variables: DEPEND, HDEPEND, > TDEPEND. DEPEND and the HDEPEND/TDEPEND pair are mutually exclusive (if > DEPEND is set, neither of the others may be, and if HDEPEND or TDEPEND is > set, DEPEND may not be). Thus current ebuilds can continue to use the simple > DEPEND, but if a maintainer wants to support cross compilation, he can > simply split DEPEND into T- and H-. Seems like the most simple conversion > path. Sacrilegious as it might be, there is another more aggressive path we might take. If we really want to support cross-compiling properly from EAPI-N+1 onwards, we could just deprecate DEPEND entirely, and support *only* BDEPEND and HDEPEND for ebuilds using the N+1 EAPI. This would make the distinction explicit, and force people to think about the issue when modernizing packages. After all, to properly support cross compiling with accurate dependency handling, practically all ebuilds need some sort of support in their dependency structure; ignoring the problem entirely only works for a minority of packages. Of course, the downsides are obvious. Thoughts?
You're assuming that dependencies will be specified correctly if we do this...
As Ciaran shortly summarized, you are missing a few key points. Cross-compilation is not that easy to understand, and I doubt many developers will be willing to maintain an extra testing environment for it. Without proper testing, even good-willing developers are likely to commit mistaken deps. Some developers will simply get angry for you adding to their burden, and just move deps randomly into one of the variables. The result will be that many packages will 'look like' supporting cross-compilation but actually be completely broken. So instead of having clear distinction between packages that were modified to support cross-compile (and hopefully weren't broken yet) and packages that weren't, you get a big bowl of random commits.
Fair points. I'm not saying we should do this, just throwing the idea out there.
Rejected (in this form) from EAPI 6: http://www.gentoo.org/proj/en/council/meeting-logs/20140617.txt
(In reply to Ulrich Müller from comment #147) > Rejected (in this form) from EAPI 6: > http://www.gentoo.org/proj/en/council/meeting-logs/20140617.txt That's unfortunate. I had been preparing a large response to this that explained that I have been doing a lot of cross-compiling lately and while it'll never be perfect, I believe we are closer to something workable than some devs here seem to think. I decided that I could put more weight behind my argument if I became a dev first and actually requested to be mentored yesterday. I was unaware of this discussion going on in the background. Having said that, I didn't expect this to be accepted as-is so if more thought is required as creffett stated then I will ensure that this happens once I am settled as a dev.
This was defeated with 3 nays and 4 abstentions, because some council members want to see a more complete solution, others were concerned that the topic didn't get much traffic on mailing lists lately. It may still be possible to get this into EAPI 6, but only if somebody will reopen discussion in the gentoo-dev ML and reach consensus. However, if nobody will pick up the ball, then it will have to wait for a later EAPI. For reference, here is the relevant part of the 20140617 council log: [15:35:39] <rich0> next 4e - HDEPEND: host dependencies for cross-compilation [15:35:39] <rich0> bug #317337 [15:35:42] <willikins> rich0: https://bugs.gentoo.org/317337 "[Future EAPI]: HDEPEND for classifying build time dependencies as host or target ones"; Gentoo Hosted Projects, PMS/EAPI; CONF; ambrop7:pms-bugs [15:35:46] <dilfridge> sigh [15:36:23] <creffett|irssi> I'm just going to go ahead and pre-abstain on this one... [15:36:54] <scarabeus> :D [15:37:03] <rich0> I don't have a problem with it. [15:37:04] -*- scarabeus is not exactly convinced by this proposal [15:37:08] <ulm> there is a proof of concept implementation in portage for this one [15:37:10] <rich0> It didn't get much list traffic though. [15:37:39] <rich0> Low list traffic = low interest... [15:37:48] <dilfridge> the distinction makes sense... but the problem is once again that it increases ebuild complexity, and most devs dont care about crosscompilation [15:38:01] <ulm> rich0: there's some recent activity in the bug thouhg [15:38:04] <ulm> *though [15:38:04] <rich0> True [15:38:05] <scarabeus> it also demands rewrite of everything [15:38:18] <scarabeus> so the dep change should be thought more of and include all possible cases [15:38:27] <scarabeus> as in "we are rewriting everything so make it damn count" [15:38:29] <rich0> Well, people could still blindly stick stuff in DEPEND, and at least it gives a way to fix things for those who do care about cross-compiling. [15:39:15] <rich0> scarabeus: this is the classic partial solution dilema. Do we make things worse by making them only a little better? [15:39:29] <blueness> rich0, correct it would mean you have DEPENDS in there that you don't necessarily need but could [15:39:44] -*- ulm points to https://wiki.gentoo.org/wiki/Future_EAPI/New_dependency_types [15:39:53] <dilfridge> Ø [15:40:50] -*- creffett|irssi thinks that DEPEND changes should be thought through top-down if we're going to do them [15:40:51] -*- scarabeus still thinks when we overhaul we should do it properly [15:40:53] <scarabeus> not just one thing [15:40:54] <creffett|irssi> ^^ [15:41:00] <ulm> can we just vote? EAPI 6 will be reiterated in any case [15:41:09] <creffett|irssi> if we're going to change the deps, let's actually think it through first, not just slap dep types on [15:41:18] <rich0> ok, are we fairly settled? [15:41:20] <dilfridge> that makes sense [15:41:36] <rich0> Ok, vote - 4e - HDEPEND: [15:41:39] -*- rich0 abstains [15:41:43] -*- ulm abstain [15:41:45] -*- dilfridge no [15:41:48] -*- blueness no [15:42:01] -*- creffett|irssi abstain [15:42:12] <rich0> dberkholz, scarabeus? [15:42:27] <dberkholz> abstain [15:42:52] <scarabeus> no [15:42:58] <rich0> Ok, defeated 0-3 with 4 abstentions. [15:43:00] <dberkholz> i think the concept is useful but am not convinced by the implementation [15:43:15] <scarabeus> no for me in this state simply :) [15:43:18] <scarabeus> but the idea is solid
I don't know what the timetable is for EAPI 6 but I don't wish to hold things up. Becoming a dev will take time, as will crafting a better solution, and if that means waiting for 7 then so be it.
James: IF you have a proper solution to this, i.e. one that solves the problem in its entirety rather than only patching the most obvious aspect of it (which is what the council fears for the current solution), I don't think not-yet being a dev will hold you back. EAPI 6 will take a while to get finalized; if you can develop a proposal we can all get behind soon-ish, it will probably be seriously considered for EAPI 6, whether you are a full dev or not.
Thank you, Ruud. In truth, I don't have a solution to hand. I was really looking to all of you to reach some kind of consensus. HDEPEND was proving adequate for me but I would have been just as happy with something else as long as it was able to fill this gap. I can understand the desire to take the wider picture into account with a total revamp. It's confusing though as some people are calling for what would most likely be a more complex solution, and yet even some of those same people are saying that anything too complex risks being misunderstood by developers. I hope these two things aren't mutually exclusive. In terms of everything needing to change to make way for HDEPEND, I don't really agree. Having cross-compiled some noteworthy targets using some tweaks (LibreOffice, Samba 3, MySQL, GTK+ 2/3, qt4 and friends), I have found that only a handful of packages really need HDEPEND. Because Portage installs all of DEPEND to / by default, the simpler form usually suffices. I will continue to become a dev as it's something I should have done 10 years ago. I will also try to collate all the views and concerns on this page into something solid.
(wrt Ruud Koolen from comment #151) > .. a proper solution to this, i.e. one that solves the > problem in its entirety rather than only patching the most obvious aspect of > it (which is what the council fears for the current solution) The only problem with the current solution as implemented by Ambroz in this bug, is the nomenclature issue, raised in comment #125 Please see also comment #130 and comment #132 -- essentially we need to stay consistent with CBUILD/CHOST/CTARGET (no-one but toolchain should worry about the last.) If you stipulate HDEPEND are runtime CHOST (ie: "embedded target") buildtime-deps that are not actually required at runtime, but must be installed for the target ROOT before the package can build, then you're there. Clearly the major usage would be headers, for which HDEPEND makes sense too. In code terms, that simply means keeping what Ambroz has, and using HDEPEND to mean what he's labelled "TDEPEND", and keep DEPEND for CBUILD (what he's labelled "HDEPEND".) The only wrinkle would be how to specify recursive dependencies of the same sort, for eg: boost I think it was, which has been raised more recently in bug 392239 -- you could either use CDEPEND or sub-slot operators. However you define the syntax, the package-manager is going to have to deal with the HDEPENDs (with the new meaning) in the context of the ebuild currently being installed, to resolve this, and what mgorny is after there. (Whatever name you give it: it's just cleaner to have a name everyone can refer to know exactly what is meant, without redefining DEPEND on an EAPI basis.) So for the example raised in this bug we'd have (as per-Spanky's comment #130): DEPEND=doxygen HDEPEND=xcb-proto RDEPEND=libXau ..and both doxygen and xcb-proto would be candidates for removal by --depclean --with-bdeps=y as far as x11-libs/libxcb is concerned. HDEPEND can be read by toolchain folks as "CHOST DEPEND" and by everyone else as "Header DEPEND" until they have another use-case for it, which is going to be esoteric, as is all of this by comparison to LIB_DEPEND/LDEPEND (a set currently referred to by various terms, depending on who you're talking to and which bit they're coding on that day.) ISTR there was one that wasn't a header, but don't recall what.
This has been discussed multiple times between different developers, and I've discussed this with ulm again, and I would like to make a few key points: 1. This is feasible for EAPI 7, if somebody is willing to carry it to the end. 2. We'll go for BDEPEND (for CBUILD) + DEPEND (for CHOST). This will be probably the most consistent, and require least changes. Please cease all further discussion on how things should be named because it is not productive and does not really matter. 3. This bug is purely about this new dependency type. Please cease all discussion on whether we should have another variable or dependency labels, or whatever. If you want that, use a dedicated PMS item for it. If labels go into EAPI 7, BDEPEND will became one. If they don't, it will be BDEPEND. End of story. 4. Cross-awareness (much like prefix-awareness) will be optional for developers. Writing ebuilds with only RDEPEND and DEPEND is still valid. Cross-aware people (possibly a dedicated team) can move appropriate deps to BDEPEND at their leisure. 5. Cross-awareness will be optioanl for the Package Manager. In particular, EAPI 7 will not suddenly require PMs to support cross-builds. 6. For non-cross builds, the Package Manager will install both DEPEND and BDEPEND to the ROOT. 7. What needs to be done is defining ROOT and some new variable with similar purpose. One of them will be the location where DEPEND (CHOST) stuff is installed, the other where BDEPEND (CBUILD) stuff is installed. I know crossdev uses some kind of SYSROOT but I have no clue which one it is, and how it relates to ROOT. Unless I'm missing something, this is all that needs to go into PMS.
1-6 sounds (In reply to Michał Górny from comment #154) > This has been discussed multiple times between different developers, and > I've discussed this with ulm again, and I would like to make a few key > points: 1-6 sounds great. I'm not sure what you mean by 7, could you give an example? crossdev sets SYSROOT to /usr/${CHOST} in cross-emerge and then sets PORTAGE_CONFIGROOT to the same value. I think ROOT is then implied from that. I know that PORTAGE_CONFIGROOT and ROOT are Portage variables but when I tried to find out what SYSROOT is actually used for, the only notable reference I could find was in cross-pkg-config. I have generally assumed that build tools will either be in the PATH or, failing that, in /. haubi pointed out in bug #509568 that the build environment itself may be prefixed but I commented that such a setup surely leads to the gates of madness. Cross-compiling is complicated enough already so I don't think we should waste too much time on corner cases like this. It's worth mentioning that crossdev assumes a ROOT of /usr/${CHOST} because that's where the toolchains it generates are configured to look. Trying to set ROOT elsewhere inevitably leads to failure unless you create toolchain wrappers that add a sysroot argument (not unlike how we create distcc wrappers). Calling econf with --sysroot (when the configure script supports it) also helps very much but these are things that can be dealt with outside of this discussion. Just wanted to add some background info.
(In reply to James Le Cuirot from comment #155) > crossdev sets SYSROOT to /usr/${CHOST} in cross-emerge and then sets > PORTAGE_CONFIGROOT to the same value. I think ROOT is then implied from > that. Correction, crossdev sets ROOT in /usr/${CHOST}/etc/portage/make.conf.
Ok, then I guess we can disregard SYSROOT and go forward with a clean variable. I'd suggest BROOT for consistency. So the idea would be that: CHOST -> DEPEND, are installed in ROOT CBUILD -> BDEPEND, are installed in BROOT For non-cross case, CHOST == CBUILD, ROOT == BROOT and both DEPEND and BDEPEND are installed. For cross case, BDEPEND packages are built for CBUILD and installed in BROOT, while DEPEND are built for CHOST and installed in ROOT. Probably the PATH contains BROOT paths first, then ROOT but I'd say that's outside of PMS scope. Does that sound fine? If so, I think we can add it to EAPI 7.
(In reply to Michał Górny from comment #157) > CHOST -> DEPEND, are installed in ROOT > CBUILD -> BDEPEND, are installed in BROOT Sounds good. I guess you would therefore also have BPREFIX? This was the variable discussed in bug #509568. > For cross case, BDEPEND packages are built for CBUILD and installed in > BROOT, while DEPEND are built for CHOST and installed in ROOT. Probably the > PATH contains BROOT paths first, then ROOT but I'd say that's outside of PMS > scope. In my experience, you don't add anything under ROOT to the executable PATH. I can see why you might think this could be useful when the executable is non-native code, but I have not found it to be necessary in practise. Include and library paths are a different matter but trying to control these directly with variables (CPPFLAGS, LDFLAGS) rarely works. --sysroot is far more effective. What I said earlier about the gates of madness wrt bug #509568 was somewhat confused as it's been a while. Best read that bug report in full to see what I really meant. BPREFIX should deal with it sufficiently.
Oh, one very important point. PMS currently states that you cannot reference ROOT in any of the src_* phases. This isn't practical when cross-compiling. If you don't pass --sysroot=${ROOT} to configure, for example, libtool will always mess up relinking. The details of how we do this in practise can be dealt with later but this restriction really needs to be lifted.
(In reply to James Le Cuirot from comment #158) > (In reply to Michał Górny from comment #157) > > CHOST -> DEPEND, are installed in ROOT > > CBUILD -> BDEPEND, are installed in BROOT > > Sounds good. I guess you would therefore also have BPREFIX? This was the > variable discussed in bug #509568. No, I don't think we actually need that. BROOT should suffice. The packages built for BDEPEND need to be considered in two contexts: 1. when they are built, the build is 'local' to the CBUILD host. During the build, CHOST == CBUILD, ROOT == BROOT, and appropriate EPREFIX is respected. 2. When the final package is built, BROOT contains the path to wherever the built executables landed. In particular, the EPREFIX from step 1 should be include in it. I don't think we really need to split this path into 'root-root' and 'build-EPREFIX'. Of course, the PM could define some kind of BPREFIX variable as implementation detail. However, I don't think PMS need to care about it. In other words: EROOT = ROOT + PREFIX for CHOST BROOT = ROOT + PREFIX for CBUILD
(In reply to Michał Górny from comment #160) > No, I don't think we actually need that. BROOT should suffice. > > 2. When the final package is built, BROOT contains the path to wherever the > built executables landed. In particular, the EPREFIX from step 1 should be > include in it. I don't think we really need to split this path into > 'root-root' and 'build-EPREFIX'. Okay, I'll buy that.
(In reply to Michał Górny from comment #154) > 2. We'll go for BDEPEND (for CBUILD) + DEPEND (for CHOST). This will be > probably the most consistent, and require least changes. No it's inconsistent, as SpanKy outlined and I expanded on. As for BROOT, don't make me laugh. > Please cease all > further discussion on how things should be named because it is not > productive and does not really matter. Yes it really does. Suddenly DEPEND means different things according to EAPI. That you think nomenclature does not matter, is a major problem, which disqualifies your from leading this task, imo. Please, let someone who knows what they're doing, like blueness, zorry or someone else from the toolchain herd, handle it. At least then we won't have to keep explaining the basics over and over again, only for someone to act like the points were never made, apparently because they still don't understand them.
To be clear: all we need is HDEPEND for CHOST dependencies in ROOT which are not runtime-required, but must be present at build-time (like headers.) The rest is gibber, afaic. Sorry to be the bearer of bad news.
(In reply to Steve L from comment #162) > (In reply to Michał Górny from comment #154) > > 2. We'll go for BDEPEND (for CBUILD) + DEPEND (for CHOST). This will be > > probably the most consistent, and require least changes. > > No it's inconsistent, as SpanKy outlined and I expanded on. > > As for BROOT, don't make me laugh. I think you've misinterpreted what SpanKy said. (In reply to SpanKY from comment #130) > so with a BDEPEND name, the natural split would be: > BDEPEND=doxygen DEPEND='xcb-proto libXau' RDEPEND=libXau > > while with a HDEPEND name, the split would be: > DEPEND=doxygen HDEPEND='xcb-proto libXau' RDEPEND=libXau > > what would be wrong would be to use: > HDEPEND=doxygen DEPEND='xcb-proto libXau' RDEPEND=libXau He only took issue with the last one, which is what was implemented originally. We are proposing the first one, which is the same thing under a less confusing name.
@Steve L, I hope you are satisfied by the work of 'someone who knows what they're doing' here. Removing from next EAPI due to no progress.
In bug 563034, Daniel Robbins has expressed a desire for a dependency type which stipulates that the same version of a particular package be installed in both ROOTs.
*** Bug 590456 has been marked as a duplicate of this bug. ***
*** Bug 611994 has been marked as a duplicate of this bug. ***
*** Bug 573306 has been marked as a duplicate of this bug. ***
So after an intense thinking session, we've came to the following table. =================== ============== ============== ============ autotools var CBUILD CHOST ------------------- -------------- --------------------------- arch native target ------------------- -------------- --------------------------- dep type BDEPEND DEPEND RDEPEND ------------------- -------------- -------------- ------------ unprefixed path / SYSROOT ROOT ------------------- -------------- -------------- ------------ relevant prefix BROOT EPREFIX ------------------- -------------- --------------------------- prefixed path BROOT ESYSROOT EROOT ------------------- -------------- -------------- ------------ autoconf var --- --with-sysroot n/a ------------------- -------------- -------------- ------------ has_version --build-root --host-root (none) or --bdepend or --depend (--rdepend?) =================== ============== ============== ============ Explanation/rationale: 1. We should use CBUILD/CHOST triplets for explanations to avoid ambiguity. 2. BDEPEND (for C*B*UILD) is added to provide dependencies that are CBUILD native. DEPEND is used for CHOST to avoid having to rename everything for CHOST deps while it's the more common of dependency types. In other words, people who care about cross will be moving deps from DEPEND to BDEPEND. 3. BDEPEND are installed into / root (but see below). SYSROOT is the base (unprefixed) location where DEPEND is installed, much like ROOT is where RDEPEND are installed. Both variables are valid only in the phase functions where the specific dep type is satisfied. 4. We support using two different prefix values simultaneously -- BROOT (or BPREFIX -- we don't do both vars since they would be equal) for CBUILD, and EPREFIX for CHOST. We do not allow separate prefixes for SYSROOT and ROOT since that would cause path conflicts (e.g. pkg-config files from SYSROOT can specify install paths that will be used in ROOT). 4a. The fun starts when using BROOT=/, SYSROOT=/, ROOT!=/, EPREFIX!=/ which renders ESYSROOT invalid. However, it's unclear if we really want to care about that, given that this kind of install was never really guaranteed to work correctly. 5. We define ESYSROOT for consistency with EROOT. 6. For convenience, we may pass --with-sysroot to autoconf+libtool build systems (the switch is defined by libtool). Other packages will have to figure out how to pass --sysroot to the compiler themselves. 6a. We may also want to define PKG_CONFIG_SYSROOT_DIR to avoid boilerplate. 7. has_version --host-root is a messy name, and not very well defined between CBUILD and CHOST at the moment, so I think we can ignore its behavior now and define it better for the future EAPI. We should probably also restrict calling them to the phases when people can actually rely on the results (dependencies). 7a. idea 1: --build-root (for CBUILD), --host-root (for CHOST DEPEND) -- still kinda sucks since RDEP also applies to CHOST; 7b. idea 2: --bdepend, --depend, [--rdepend] -- using dependency types should make it quite clear when it's valid although it implies changing current syntax; 7c. idea 3: --broot, --sysroot, [--root] -- using path variable names, except that BROOT is really kind of EPREFIX/EROOT var.
(In reply to Michał Górny from comment #170) > So after an intense thinking session, we've came to the following table. > > =================== ============== ============== ============ > autotools var CBUILD CHOST > ------------------- -------------- --------------------------- > arch native target > ------------------- -------------- --------------------------- > dep type BDEPEND DEPEND RDEPEND > ------------------- -------------- -------------- ------------ > unprefixed path / SYSROOT ROOT > ------------------- -------------- -------------- ------------ > relevant prefix BROOT EPREFIX > ------------------- -------------- --------------------------- > prefixed path BROOT ESYSROOT EROOT > ------------------- -------------- -------------- ------------ > autoconf var --- --with-sysroot n/a > ------------------- -------------- -------------- ------------ > has_version --build-root --host-root (none) > or --bdepend or --depend (--rdepend?) > =================== ============== ============== ============ > > > Explanation/rationale: > > 1. We should use CBUILD/CHOST triplets for explanations to avoid ambiguity. I find that very confusing. CBUILD/CHOST being able to differ should ultimately be a side effect of having different paths. That is, you have a normal gentoo install in SYSROOT (CHOST=CBUILD) and portage figures out by itself that it should use CBUILD from /etc/portage/make.conf and not from SYSROOT. Plus, keep in mind that a big consumer of this would be catalyst builds where CHOST==CBUILD. [...] > 4. We support using two different prefix values simultaneously -- BROOT (or > BPREFIX -- we don't do both vars since they would be equal) for CBUILD, and > EPREFIX for CHOST. We do not allow separate prefixes for SYSROOT and ROOT > since that would cause path conflicts (e.g. pkg-config files from SYSROOT > can specify install paths that will be used in ROOT). SYSROOT!=ROOT should not matter at all here. ROOT is merely where the target package's files are moved. > 7. has_version --host-root is a messy name, and not very well defined > between CBUILD and CHOST at the moment, so I think we can ignore its > behavior now and define it better for the future EAPI. We should probably > also restrict calling them to the phases when people can actually rely on > the results (dependencies). > > 7a. idea 1: --build-root (for CBUILD), --host-root (for CHOST DEPEND) -- > still kinda sucks since RDEP also applies to CHOST; RDEP only applies to ROOT (see the confusion when doing CHOST/CBUILD here :) ) I do not really understand the idea of BROOT: It seems to me BROOT=${EPREFIX:=/} is the only useful case. The only way I can see that useful is for a prefix install in /home/foo/bar where you would want to make a SYSROOT in /home/foo/baz, which has probably very very few demand and can likely work with relative paths. Apart for "consistency", what is the point of SYSROOT vs ESYSROOT ? SYSROOT is where to look for files (headers, libs, etc.), so in that sense it should be an absolute path. The way it is defined, it looks like SYSROOT should never ever be used in favor of ESYSROOT in ebuilds. Which raises the question: Why ESYSROOT at all ? Note that ROOT vs EROOT distinction has a point: ROOT==/ is the proper condition for being able to execute "${EROOT}/usr/bin/foo" normally. Finally, besides those (rather minor) points, thanks for moving forward with this, and +1 on the whole thing.
(In reply to Alexis Ballier from comment #171) > (In reply to Michał Górny from comment #170) > > 4. We support using two different prefix values simultaneously -- BROOT (or > > BPREFIX -- we don't do both vars since they would be equal) for CBUILD, and > > EPREFIX for CHOST. We do not allow separate prefixes for SYSROOT and ROOT > > since that would cause path conflicts (e.g. pkg-config files from SYSROOT > > can specify install paths that will be used in ROOT). > > SYSROOT!=ROOT should not matter at all here. ROOT is merely where the target > package's files are moved. I got a little confused while discussing this point last night and I think separate prefixes might be okay. I will do some experimentation but I'm currently without my main desktop and my ARM box is a little slow. > > 7. has_version --host-root is a messy name, and not very well defined > > between CBUILD and CHOST at the moment, so I think we can ignore its > > behavior now and define it better for the future EAPI. We should probably > > also restrict calling them to the phases when people can actually rely on > > the results (dependencies). > > > > 7a. idea 1: --build-root (for CBUILD), --host-root (for CHOST DEPEND) -- > > still kinda sucks since RDEP also applies to CHOST; > > RDEP only applies to ROOT (see the confusion when doing CHOST/CBUILD here :) > > > > > 7b. idea 2: --bdepend, --depend, [--rdepend] -- using dependency types should make it quite clear when it's valid although it implies changing current syntax; > > > > 7c. idea 3: --broot, --sysroot, [--root] -- using path variable names, except that BROOT is really kind of EPREFIX/EROOT var. On seeing these alternatives, I like 7b and 7c better. 7b would probably be the least confusing for devs. > I do not really understand the idea of BROOT: It seems to me > BROOT=${EPREFIX:=/} is the only useful case. The only way I can see that > useful is for a prefix install in /home/foo/bar where you would want to make > a SYSROOT in /home/foo/baz, which has probably very very few demand and can > likely work with relative paths. I'm not quite following you but if the prefixes for / and ROOT can differ then it is needed. You can rely on the PATH for most tools but sometimes you need to call something in /usr/libexec for example. I talked about this in bug #509568. > Apart for "consistency", what is the point of SYSROOT vs ESYSROOT ? SYSROOT > is where to look for files (headers, libs, etc.), so in that sense it should > be an absolute path. The way it is defined, it looks like SYSROOT should > never ever be used in favor of ESYSROOT in ebuilds. Which raises the > question: Why ESYSROOT at all ? > Note that ROOT vs EROOT distinction has a point: ROOT==/ is the proper > condition for being able to execute "${EROOT}/usr/bin/foo" normally. vapier initially suggested ESYSROOT in bug #563034 for consistency. I take your point that you may never need the unprefixed form but it's hard to say for sure and having both makes it clear whether the prefix is already present or not.
(In reply to James Le Cuirot from comment #172) has_version (and best_version) is currently the only command in PMS using long options, which is somewhat unsystematic. So if we are to touch that syntax, I would strongly prefer that we take that opportunity to switch to short options (which would finally enable us to follow POSIX Guidelines which for long options is hard to implement in bash). Looks like -b, -h, and -r would be the natural choices there, maybe with the default being -r when no option is passed.
(In reply to Alexis Ballier from comment #171) > (In reply to Michał Górny from comment #170) > > So after an intense thinking session, we've came to the following table. > > > > =================== ============== ============== ============ > > autotools var CBUILD CHOST > > ------------------- -------------- --------------------------- > > arch native target > > ------------------- -------------- --------------------------- > > dep type BDEPEND DEPEND RDEPEND > > ------------------- -------------- -------------- ------------ > > unprefixed path / SYSROOT ROOT > > ------------------- -------------- -------------- ------------ > > relevant prefix BROOT EPREFIX > > ------------------- -------------- --------------------------- > > prefixed path BROOT ESYSROOT EROOT > > ------------------- -------------- -------------- ------------ > > autoconf var --- --with-sysroot n/a > > ------------------- -------------- -------------- ------------ > > has_version --build-root --host-root (none) > > or --bdepend or --depend (--rdepend?) > > =================== ============== ============== ============ > > > > > > Explanation/rationale: > > > > 1. We should use CBUILD/CHOST triplets for explanations to avoid ambiguity. > > > I find that very confusing. CBUILD/CHOST being able to differ should > ultimately be a side effect of having different paths. That is, you have a > normal gentoo install in SYSROOT (CHOST=CBUILD) and portage figures out by > itself that it should use CBUILD from /etc/portage/make.conf and not from > SYSROOT. > Plus, keep in mind that a big consumer of this would be catalyst builds > where CHOST==CBUILD. Stop substituting values for strings that are not variables. I'm merely talking of terms used in PMS so that people have any clue which is which. If you have a better alternative, then please suggest it instead of complaining about the only unambiguous form I know of. > [...] > > 4. We support using two different prefix values simultaneously -- BROOT (or > > BPREFIX -- we don't do both vars since they would be equal) for CBUILD, and > > EPREFIX for CHOST. We do not allow separate prefixes for SYSROOT and ROOT > > since that would cause path conflicts (e.g. pkg-config files from SYSROOT > > can specify install paths that will be used in ROOT). > > SYSROOT!=ROOT should not matter at all here. ROOT is merely where the target > package's files are moved. You are confusing things. All this says is that EPREFIX has to be the same within SYSROOT and ROOT. > > 7. has_version --host-root is a messy name, and not very well defined > > between CBUILD and CHOST at the moment, so I think we can ignore its > > behavior now and define it better for the future EAPI. We should probably > > also restrict calling them to the phases when people can actually rely on > > the results (dependencies). > > > > 7a. idea 1: --build-root (for CBUILD), --host-root (for CHOST DEPEND) -- > > still kinda sucks since RDEP also applies to CHOST; > > RDEP only applies to ROOT (see the confusion when doing CHOST/CBUILD here :) > ) That's what the comment is saying... > I do not really understand the idea of BROOT: It seems to me > BROOT=${EPREFIX:=/} is the only useful case. The only way I can see that > useful is for a prefix install in /home/foo/bar where you would want to make > a SYSROOT in /home/foo/baz, which has probably very very few demand and can > likely work with relative paths. The idea is that you might be running a Gentoo Prefix system, and building for another Prefix. For example, you may be running a Gentoo Prefix on another distribution, and cross-compiling for Android which requires installing inside a Prefix. > Apart for "consistency", what is the point of SYSROOT vs ESYSROOT ? SYSROOT > is where to look for files (headers, libs, etc.), so in that sense it should > be an absolute path. The way it is defined, it looks like SYSROOT should > never ever be used in favor of ESYSROOT in ebuilds. Which raises the > question: Why ESYSROOT at all ? In some cases you will get paths containing EPREFIX already, e.g. when querying pkg-config. In this case, you need to use 'pure' SYSROOT to avoid double EPREFIX.
(In reply to Ulrich Müller from comment #173) > (In reply to James Le Cuirot from comment #172) > > has_version (and best_version) is currently the only command in PMS using > long options, which is somewhat unsystematic. So if we are to touch that > syntax, I would strongly prefer that we take that opportunity to switch to > short options (which would finally enable us to follow POSIX Guidelines > which for long options is hard to implement in bash). > > Looks like -b, -h, and -r would be the natural choices there, maybe with the > default being -r when no option is passed. Using -h will be confusing since it doesn't correspond with anything, really. As James pointed out, the variants based on dependency types will probably be the least confusing to developers. In which case I'd go for -b [BDEPEND], -d [DEPEND], -r [RDEPEND]. I know it's not perfect but I think it'd be less confusing. That said, maybe after all it would be best to define dependency types first, and then define everything else (e.g. prefixes) in terms of dependency types. That should further reduce confusion.
(In reply to Michał Górny from comment #174) > (In reply to Alexis Ballier from comment #171) > > (In reply to Michał Górny from comment #170) > > > So after an intense thinking session, we've came to the following table. > > > > > > =================== ============== ============== ============ > > > autotools var CBUILD CHOST > > > ------------------- -------------- --------------------------- > > > arch native target > > > ------------------- -------------- --------------------------- > > > dep type BDEPEND DEPEND RDEPEND > > > ------------------- -------------- -------------- ------------ > > > unprefixed path / SYSROOT ROOT > > > ------------------- -------------- -------------- ------------ > > > relevant prefix BROOT EPREFIX > > > ------------------- -------------- --------------------------- > > > prefixed path BROOT ESYSROOT EROOT > > > ------------------- -------------- -------------- ------------ > > > autoconf var --- --with-sysroot n/a > > > ------------------- -------------- -------------- ------------ > > > has_version --build-root --host-root (none) > > > or --bdepend or --depend (--rdepend?) > > > =================== ============== ============== ============ > > > > > > > > > Explanation/rationale: > > > > > > 1. We should use CBUILD/CHOST triplets for explanations to avoid ambiguity. > > > > > > I find that very confusing. CBUILD/CHOST being able to differ should > > ultimately be a side effect of having different paths. That is, you have a > > normal gentoo install in SYSROOT (CHOST=CBUILD) and portage figures out by > > itself that it should use CBUILD from /etc/portage/make.conf and not from > > SYSROOT. > > Plus, keep in mind that a big consumer of this would be catalyst builds > > where CHOST==CBUILD. > > Stop substituting values for strings that are not variables. I'm merely > talking of terms used in PMS so that people have any clue which is which. If > you have a better alternative, then please suggest it instead of complaining > about the only unambiguous form I know of. That's exactly what I wrote: CHOST/CBUILD/etc have nothing to do with this proposal and should be left alone. Current PMS description of those is perfectly fine. What this is all about is paths where to pull dependencies and how to use them. See below about what confusion this causes. > > [...] > > > 4. We support using two different prefix values simultaneously -- BROOT (or > > > BPREFIX -- we don't do both vars since they would be equal) for CBUILD, and > > > EPREFIX for CHOST. We do not allow separate prefixes for SYSROOT and ROOT > > > since that would cause path conflicts (e.g. pkg-config files from SYSROOT > > > can specify install paths that will be used in ROOT). > > > > SYSROOT!=ROOT should not matter at all here. ROOT is merely where the target > > package's files are moved. > > You are confusing things. All this says is that EPREFIX has to be the same > within SYSROOT and ROOT. I'm not following here: Aren't SYSROOT and ROOT meant *not* to contain anything about EPREFIX ? So what you mean is ESYSROOT and EROOT will get the same EPREFIX prepended ? That's very fine then. > > > 7. has_version --host-root is a messy name, and not very well defined > > > between CBUILD and CHOST at the moment, so I think we can ignore its > > > behavior now and define it better for the future EAPI. We should probably > > > also restrict calling them to the phases when people can actually rely on > > > the results (dependencies). > > > > > > 7a. idea 1: --build-root (for CBUILD), --host-root (for CHOST DEPEND) -- > > > still kinda sucks since RDEP also applies to CHOST; > > > > RDEP only applies to ROOT (see the confusion when doing CHOST/CBUILD here :) > > ) > > That's what the comment is saying... Your comment says it applies to CHOST. From what I'm able to understand is that you're using CHOST interchangeably for SYSROOT or ROOT, which one which comment applies to is still unclear to me. Plus, CHOST==CBUILD is a perfectly fine usage of ROOT and SYSROOT (catalyst), so this confuses me even more. > > I do not really understand the idea of BROOT: It seems to me > > BROOT=${EPREFIX:=/} is the only useful case. The only way I can see that > > useful is for a prefix install in /home/foo/bar where you would want to make > > a SYSROOT in /home/foo/baz, which has probably very very few demand and can > > likely work with relative paths. > > The idea is that you might be running a Gentoo Prefix system, and building > for another Prefix. For example, you may be running a Gentoo Prefix on > another distribution, and cross-compiling for Android which requires > installing inside a Prefix. So, in that case, BROOT would be the host's EPREFIX and EPREFIX would be the android prefix ? Confusing, but prefix is already very confusing. Lacking a better idea, that seems fine then. Thanks. > > Apart for "consistency", what is the point of SYSROOT vs ESYSROOT ? SYSROOT > > is where to look for files (headers, libs, etc.), so in that sense it should > > be an absolute path. The way it is defined, it looks like SYSROOT should > > never ever be used in favor of ESYSROOT in ebuilds. Which raises the > > question: Why ESYSROOT at all ? > > In some cases you will get paths containing EPREFIX already, e.g. when > querying pkg-config. In this case, you need to use 'pure' SYSROOT to avoid > double EPREFIX. Would make sense. What I fear though is all the tools that expect SYSROOT would need to be patched for prefix unless one can 'export SYSROOT=$ESYSROOT', but I would expect the variable to be read only.
Yes to all your final conclusions; I think we're on the same page here now. (In reply to Alexis Ballier from comment #176) > Would make sense. What I fear though is all the tools that expect SYSROOT > would need to be patched for prefix unless one can 'export > SYSROOT=$ESYSROOT', but I would expect the variable to be read only. How many tools are realistically taking 'SYSROOT' variable from the environment? From what I gathered so far, it seemed like most of the time we'd actually have to pass it somehow via command-line options.
(In reply to Michał Górny from comment #177) > Yes to all your final conclusions; I think we're on the same page here now. > > (In reply to Alexis Ballier from comment #176) > > Would make sense. What I fear though is all the tools that expect SYSROOT > > would need to be patched for prefix unless one can 'export > > SYSROOT=$ESYSROOT', but I would expect the variable to be read only. > > How many tools are realistically taking 'SYSROOT' variable from the > environment? From what I gathered so far, it seemed like most of the time > we'd actually have to pass it somehow via command-line options. A quick grep in /usr/bin shows crossdev, apr1-config, apu1-config and freetype-config; the -config ones are likely something we inject ourselves so should be fine /usr/share is more worrysome: /usr/share/aclocal/gpg-error.m4: case "${SYSROOT}" in /usr/share/aclocal/gpg-error.m4: if test -x "${SYSROOT}/bin/gpg-error-config" ; then /usr/share/aclocal/libgcrypt.m4: case "${SYSROOT}" in /usr/share/aclocal/libgcrypt.m4: if test -x "${SYSROOT}/bin/libgcrypt-config" ; then /usr/share/aclocal/gpgme.m4: case "${SYSROOT}" in /usr/share/aclocal/gpgme.m4: if test -x "${SYSROOT}/bin/gpgme-config" ; then /usr/share/elt-patches/sys-lib-dlsearch/2.4: - search $SYSROOT/etc/ld.so.conf /usr/share/elt-patches/sys-lib-dlsearch/2.4:+ if test -f "$SYSROOT"/etc/ld.so.conf; then
(In reply to Michał Górny from comment #177) > Yes to all your final conclusions; I think we're on the same page here now. > > (In reply to Alexis Ballier from comment #176) > > Would make sense. What I fear though is all the tools that expect SYSROOT > > would need to be patched for prefix unless one can 'export > > SYSROOT=$ESYSROOT', but I would expect the variable to be read only. > > How many tools are realistically taking 'SYSROOT' variable from the > environment? From what I gathered so far, it seemed like most of the time > we'd actually have to pass it somehow via command-line options. I initially used "SYSROOT" in my cross-boss project and the only thing I collided with was Qt4, which resets it to a different value. I ended up using CB_SYSROOT instead. This is a legitimate concern, perhaps we should use something else.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=02df0addf8f8477590176807d747b4cbb57b7c32 commit 02df0addf8f8477590176807d747b4cbb57b7c32 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2018-03-16 08:30:41 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2018-03-17 21:43:03 +0000 Support BDEPEND (CBUILD dependencies) in EAPI 7 Bug: https://bugs.gentoo.org/317337 Closes: https://github.com/gentoo/portage/pull/268 Reviewed-by: Zac Medico <zmedico@gentoo.org> bin/eapi.sh | 4 +++ bin/ebuild.sh | 29 ++++++++++++++++------ bin/phase-functions.sh | 2 +- pym/_emerge/Package.py | 5 ++-- pym/_emerge/actions.py | 1 + pym/_emerge/depgraph.py | 10 +++++++- pym/portage/__init__.py | 2 +- pym/portage/cache/metadata.py | 2 +- pym/portage/dbapi/bintree.py | 7 +++--- pym/portage/dbapi/porttree.py | 2 +- pym/portage/dbapi/vartree.py | 4 +-- pym/portage/dep/_slot_operator.py | 7 ++++-- pym/portage/eapi.py | 7 +++++- .../package/ebuild/_config/special_env_vars.py | 3 ++- pym/portage/package/ebuild/config.py | 2 +- 15 files changed, 62 insertions(+), 25 deletions(-)}
*** Bug 654032 has been marked as a duplicate of this bug. ***
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/pms.git/commit/?id=d1ea951e810488d3047be855f7f06d343c1cc68a commit d1ea951e810488d3047be855f7f06d343c1cc68a Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-23 20:53:59 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7 drops --host-root in favor of -b, -d, and -r Bug: https://bugs.gentoo.org/317337 eapi-differences.tex | 8 +++++--- pkg-mgr-commands.tex | 30 ++++++++++++++++++++---------- 2 files changed, 25 insertions(+), 13 deletions(-) https://gitweb.gentoo.org/proj/pms.git/commit/?id=27a0bf1961ae56fd6b948d5ffea9a9e4ae35fd91 commit 27a0bf1961ae56fd6b948d5ffea9a9e4ae35fd91 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-17 21:39:27 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7 has BROOT Bug: https://bugs.gentoo.org/317337 eapi-differences.tex | 4 ++++ ebuild-env-vars.tex | 20 ++++++++++++++------ 2 files changed, 18 insertions(+), 6 deletions(-) https://gitweb.gentoo.org/proj/pms.git/commit/?id=1404b78d0bc62ba9c3038e91b240ec13ffd3744d commit 1404b78d0bc62ba9c3038e91b240ec13ffd3744d Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2017-10-04 16:19:16 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7 has BDEPEND Bug: https://bugs.gentoo.org/317337 dependencies.tex | 33 ++++++++++++++++++++++++++++++--- eapi-differences.tex | 4 ++++ ebuild-vars.tex | 1 + eclasses.tex | 10 +++++----- metadata-cache.tex | 3 ++- 5 files changed, 42 insertions(+), 9 deletions(-) https://gitweb.gentoo.org/proj/pms.git/commit/?id=283a85538403bcb1b4d03a3b6178835e4129f826 commit 283a85538403bcb1b4d03a3b6178835e4129f826 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-19 21:36:40 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7: econf adds --with-sysroot Bug: https://bugs.gentoo.org/317337 eapi-differences.tex | 3 ++- pkg-mgr-commands.tex | 17 +++++++++++------ 2 files changed, 13 insertions(+), 7 deletions(-) https://gitweb.gentoo.org/proj/pms.git/commit/?id=8d9ba58d1433ea80f28f31fbd662ad0b0a83ad87 commit 8d9ba58d1433ea80f28f31fbd662ad0b0a83ad87 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-17 21:23:58 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7 has SYSROOT and ESYSROOT Bug: https://bugs.gentoo.org/317337 eapi-differences.tex | 4 ++++ ebuild-env-vars.tex | 34 +++++++++++++++++++++++++--------- 2 files changed, 29 insertions(+), 9 deletions(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=d51ca86a44fd10b13db749a7945bba1573b76ff5 commit d51ca86a44fd10b13db749a7945bba1573b76ff5 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-28 22:11:17 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-01 08:28:43 +0000 econf: Pass --with-sysroot="${ESYSROOT:-/}" under EAPI 7 Bug: https://bugs.gentoo.org/317337 bin/eapi.sh | 4 ++++ bin/phase-helpers.sh | 8 +++++++- 2 files changed, 11 insertions(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=d8d47bf451bb3fd6bce1cdb035a5f12253f5a167 commit d8d47bf451bb3fd6bce1cdb035a5f12253f5a167 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2017-09-30 20:45:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-01 08:28:43 +0000 Export BROOT to ebuild env in EAPI 7 Export the BROOT variable corresponding to the location where BDEPEND are installed. Bug: https://bugs.gentoo.org/317337 bin/eapi.sh | 4 ++++ bin/ebuild.sh | 3 +++ bin/phase-functions.sh | 3 +++ man/ebuild.5 | 6 ++++++ pym/portage/eapi.py | 7 ++++++- pym/portage/package/ebuild/_config/LocationsManager.py | 4 ++++ pym/portage/package/ebuild/_config/special_env_vars.py | 4 ++-- pym/portage/package/ebuild/config.py | 10 +++++++++- repoman/cnf/qa_data/qa_data.yaml | 2 +- repoman/man/repoman.1 | 2 +- .../pym/repoman/modules/linechecks/assignment/assignment.py | 9 ++++++++- repoman/pym/repoman/modules/linechecks/quotes/quotes.py | 3 ++- 12 files changed, 49 insertions(+), 8 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=a41dacf7926c0242678300a8e1770df84029e8a4 commit a41dacf7926c0242678300a8e1770df84029e8a4 Author: James Le Cuirot <chewi@gentoo.org> AuthorDate: 2018-04-23 21:49:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-05-01 08:28:43 +0000 Export SYSROOT and ESYSROOT in ebuild env in EAPI 7 SYSROOT is the new destination for DEPEND, however it must currently be equal either / or ROOT. This is partly to simplify the implementation and partly because supporting a third separate destination would not be of much practical benefit anyway. Bug: https://bugs.gentoo.org/317337 bin/eapi.sh | 4 ++++ bin/phase-functions.sh | 6 ++++++ pym/_emerge/actions.py | 2 +- pym/_emerge/depgraph.py | 4 +++- pym/portage/__init__.py | 6 +++--- pym/portage/_legacy_globals.py | 3 ++- pym/portage/eapi.py | 7 ++++++- .../package/ebuild/_config/LocationsManager.py | 19 +++++++++++++++-- pym/portage/package/ebuild/config.py | 24 +++++++++++++++++++--- 9 files changed, 63 insertions(+), 12 deletions(-)
