Bug 532224 – sys-apps/portage: add @profile which is similar to @system, except that included packages must have complete dependency specifications allowing for optimal parallelization

• Status: RESOLVED FIXED
• Product: Portage Development

[Description Zac Medico 2014-12-11 02:01:33 UTC]

The motivation to have @profile separate from @system is that @system packages may have incomplete dependency specifications (due to long-standing Gentoo policy), and incomplete dependency specifications have deleterious effects on the ability of emerge --jobs to parallelize builds. So, unlike @system, packages added to @profile do not hurt emerge --jobs parallelization

[Comment 1 Zac Medico 2014-12-11 02:14:57 UTC]

I have a patch in this branch:

	https://github.com/zmedico/portage/commits/profile_package_set

I've posted it for review here:

	http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4968

[Comment 2 Zac Medico 2014-12-11 08:16:43 UTC]

This is in the master branch now:

https://github.com/gentoo/portage/commit/28828655da860324861af9adea0794a0a61196bf

[Comment 3 Zac Medico 2014-12-11 08:44:43 UTC]

@pms-bugs: This feature is hidden behind a layout.conf profile-format flag, so it's beyond the scope of PMS. Package managers should reject the profile if they
don't recognize the profile-formats flags that it declares.

[Comment 4 Ulrich Müller 2014-12-11 11:28:07 UTC]

(In reply to Zac Medico from comment #3)
> @pms-bugs: This feature is hidden behind a layout.conf profile-format flag,
> so it's beyond the scope of PMS. Package managers should reject the profile
> if they don't recognize the profile-formats flags that it declares.

So IIUC you are suggesting that portage should violate the spec (which says that dependency specifications must be prefixed by * or they are to be ignored), depending on the setting of some hidden variable in layout.conf?

Can we go for a clean solution please and make this EAPI dependent?

PMS reference:
http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-520005.2.6

[Comment 5 Ciaran McCreesh 2014-12-11 15:07:05 UTC]

profile_format is a string, not a set...

[Comment 6 Zac Medico 2014-12-11 16:20:45 UTC]

(In reply to Ulrich Müller from comment #4)
> (In reply to Zac Medico from comment #3)
> > @pms-bugs: This feature is hidden behind a layout.conf profile-format flag,
> > so it's beyond the scope of PMS. Package managers should reject the profile
> > if they don't recognize the profile-formats flags that it declares.
> 
> So IIUC you are suggesting that portage should violate the spec (which says
> that dependency specifications must be prefixed by * or they are to be
> ignored), depending on the setting of some hidden variable in layout.conf?

No, I am suggesting that it's reasonable for a package manager to reject a repository containing unrecognized flags in the layout.conf profile-formats field. Portage has supported profile profile-formats settings such as this for more than 3 years now:

https://github.com/gentoo/portage/commit/a34ac493d13ba5ad20b07f25f4e4f054e303eecb

If you'd like to track the history, search for changes to the _valid_profile_formats variable added in this commit:

https://github.com/gentoo/portage/commit/22996b1d7c7214da8f0bff76d9f814a00f58fdce

> Can we go for a clean solution please and make this EAPI dependent?
> 
> PMS reference:
> http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-520005.2.6

Absolutely, but having extensions enabled via profile-formats is not mutually exclusive. We can certainly do both.

(In reply to Ciaran McCreesh from comment #5)
> profile_format is a string, not a set...

I'm not sure if we're talking about the same thing. The profile-formats field that I'm talking about has been a *set* since it's inception in this commit:

https://github.com/gentoo/portage/commit/a34ac493d13ba5ad20b07f25f4e4f054e303eecb

[Comment 7 Ciaran McCreesh 2014-12-11 16:23:47 UTC]

(In reply to Zac Medico from comment #6)
> (In reply to Ciaran McCreesh from comment #5)
> > profile_format is a string, not a set...
> 
> I'm not sure if we're talking about the same thing. The profile-formats
> field that I'm talking about has been a *set* since it's inception in this
> commit:
> 
> https://github.com/gentoo/portage/commit/
> a34ac493d13ba5ad20b07f25f4e4f054e303eecb

Well it's a bug in your implementation if you allow that set to contain more than one value.

[Comment 8 Zac Medico 2014-12-11 16:30:21 UTC]

(In reply to Ciaran McCreesh from comment #7)
> Well it's a bug in your implementation if you allow that set to contain more
> than one value.

Well, we've had more than 3 years to sort this out now. So, this feels like obstructionism.

[Comment 9 Ciaran McCreesh 2014-12-11 16:34:19 UTC]

(In reply to Zac Medico from comment #8)
> (In reply to Ciaran McCreesh from comment #7)
> > Well it's a bug in your implementation if you allow that set to contain more
> > than one value.
> 
> Well, we've had more than 3 years to sort this out now. So, this feels like
> obstructionism.

I guess it hasn't been an issue before now, if you haven't been using this.

[Comment 10 Michał Górny 2014-12-11 16:39:33 UTC]

(In reply to Ciaran McCreesh from comment #5)
> profile_format is a string, not a set...

Could you point to the relevant specification, please?

[Comment 11 David Leverton 2014-12-12 19:44:45 UTC]

I don't have any opinion on the "profile" set itself, but didn't we establish that profile-formats is terrible when it was first introduced?

Originally, the idea was that it was a single value in layout.conf, which doesn't allow different profiles in the same repository to use different formats.  Then it was changed to a list of formats that has to cover all the formats that are used in the repository, except that still doesn't tell you which profiles use which format.  The solution to /that/ was some mumbling about how profiles in future formats will go under profiles2/, profiles3/, etc, allowing the PM to disambiguate that way (but we don't need to do that for the first "extended" format, portage-1, because... reasons), but AFAIK that wasn't ever implemented, even now that Portage supports portage-2 as well.

Then it turns out that, even though the intention was apparently that each individual profile would conform to one particular format (assumption supported by the original single-value proposal, and the names portage-1 and portage-2), Portage doesn't implement it that way, instead turning on all features that belong to any formats listed in the layout.conf, which might result in a combination that isn't supported by any single format.  This new proposal, as well as the existing "profile-bashrcs", seem to officially adopt that interpretation, so now it's a random blend of self-contained format versions and individual "mix-in" features.

All this and no tangible improvements over the profile "eapi" file - I really don't see why we need both.

[Comment 12 Tim Harder 2014-12-12 20:51:15 UTC]

(In reply to David Leverton from comment #11)
> All this and no tangible improvements over the profile "eapi" file - I
> really don't see why we need both.

I agree with this for the most part. It would be great if PMS enumerated beneficial profile format changes.

The only reason I guess portage doesn't want that is it drags out features that people might want *right now* instead of having discussions that could alter or drop what they wanted.

[Comment 13 Zac Medico 2014-12-12 22:23:02 UTC]

(In reply to David Leverton from comment #11)
> I don't have any opinion on the "profile" set itself, but didn't we
> establish that profile-formats is terrible when it was first introduced?

It's an extremely useful feature for repositories other than "gentoo" that need to introduce new profile formats for various reasons. At least "profile-formats = portage-2" appears to be widely used, according to google search results. Also, the recently added support for "profile-formats = profile-bashrcs" will be used at least by chromium-os repositories (patch was contributed by a developer from chromium.org).

> Originally, the idea was that it was a single value in layout.conf, which
> doesn't allow different profiles in the same repository to use different
> formats.  Then it was changed to a list of formats that has to cover all the
> formats that are used in the repository, except that still doesn't tell you
> which profiles use which format.  The solution to /that/ was some mumbling
> about how profiles in future formats will go under profiles2/, profiles3/,
> etc, allowing the PM to disambiguate that way (but we don't need to do that
> for the first "extended" format, portage-1, because... reasons), but AFAIK
> that wasn't ever implemented, even now that Portage supports portage-2 as
> well.

I think that for the target use-case(s) that you are thinking of, the existing profile EAPI support is the right tool. In that context, you can fork your profiles when you need to support a new profile EAPI.

Generally, profile-formats is more useful in an environment where an organization maintains its own profiles and processes of forking profiles and implementing new EAPIs are unnecessary burdens.

> All this and no tangible improvements over the profile "eapi" file - I
> really don't see why we need both.

Creating a new EAPI for each profile format change is an unnecessary burden within the context of an organization that maintains its own profiles in a separate repository.

[Comment 14 Tim Harder 2014-12-12 22:51:58 UTC]

(In reply to Zac Medico from comment #13)
> It's an extremely useful feature for repositories other than "gentoo" that
> need to introduce new profile formats for various reasons. At least
> "profile-formats = portage-2" appears to be widely used, according to google
> search results. Also, the recently added support for "profile-formats =
> profile-bashrcs" will be used at least by chromium-os repositories (patch
> was contributed by a developer from chromium.org).

One of the issues is these formats become defacto-standards as people wanting to use such profiles with other PMs ask for and require support. So the only solution third parties have in order to support such a "standard" is to read the portage man page, code, and potentially even mirror the bugs of its initial implementation.

Perhaps we should stop cramming feature after feature into single, major EAPIs and speed up the process using minor EAPI bumps like has been discussed before, e.g. EAPI 5 -> 5.1 -> 5.2 -> ... -> 6. Of course this has its downsides as well, but it could allow for supporting potentially simpler features like profile extensions much faster than waiting years between major bumps.

[Comment 15 Zac Medico 2014-12-12 23:07:22 UTC]

(In reply to Tim Harder from comment #14)
> One of the issues is these formats become defacto-standards as people
> wanting to use such profiles with other PMs ask for and require support. So
> the only solution third parties have in order to support such a "standard"
> is to read the portage man page, code, and potentially even mirror the bugs
> of its initial implementation.

I think this is a small price to pay for the ease with which profile-formats allows us to roll out enhancements that help to make Gentoo an appealing choice to organizations such as chromium.org (among many others). Drawing in users from organizations like this helps to increase the size of our community and make it stronger.

> Perhaps we should stop cramming feature after feature into single, major
> EAPIs and speed up the process using minor EAPI bumps like has been
> discussed before, e.g. EAPI 5 -> 5.1 -> 5.2 -> ... -> 6. Of course this has
> its downsides as well, but it could allow for supporting potentially simpler
> features like profile extensions much faster than waiting years between
> major bumps.

People have been talking about this for years, but it never happens.

[Comment 16 David Leverton 2014-12-14 15:40:06 UTC]

(In reply to Zac Medico from comment #13)
> Creating a new EAPI for each profile format change is an unnecessary burden
> within the context of an organization that maintains its own profiles in a
> separate repository.

Well, assuming they really don't care about multiple PM support, I don't see how defining a new private EAPI is harder than defining a new profile format (plus it would allow them to introduce new ebuild features as well if required, and it's not obvious why this kind of organisation would want to be able to do one but not the other).  And conversely, if they do want other PMs to support it then a profile format wouldn't need any /less/ paperwork than a new EAPI (which doesn't have to mean full Council approval, if it's not meant for the main Gentoo tree).

[Comment 17 Zac Medico 2014-12-14 18:30:35 UTC]

(In reply to David Leverton from comment #16)
> (In reply to Zac Medico from comment #13)
> > Creating a new EAPI for each profile format change is an unnecessary burden
> > within the context of an organization that maintains its own profiles in a
> > separate repository.
> 
> Well, assuming they really don't care about multiple PM support, I don't see
> how defining a new private EAPI is harder than defining a new profile format
> (plus it would allow them to introduce new ebuild features as well if
> required, and it's not obvious why this kind of organisation would want to
> be able to do one but not the other).

They may or may not want to do both at any given time, however, they can be considered as orthogonal concerns. Using profile-formats to mix in profile extensions with existing EAPIs is much more convenient if it happens that profile extensions are the only desired enhancement at a particular time.

Also, with profile-formats extensions, it's generally more feasible to avoid forking profiles. Clients running an old version of portage that doesn't support new profile-formats extensions simply receive a warning about the unrecognized extensions, rather than a fatal error. When they receive such a warning, they simply 'emerge --oneshot portage' at the earliest opportunity.

> And conversely, if they do want other
> PMs to support it then a profile format wouldn't need any /less/ paperwork
> than a new EAPI (which doesn't have to mean full Council approval, if it's
> not meant for the main Gentoo tree).

EAPIs cannot be mixed like profile-formats extensions can. When Gentoo releases a new EAPI, they'll be able to mix their profile extensions with it immediately,  with no need to release a new EAPI of their own to integrate features from the new Gentoo EAPI.

[Comment 18 Brian Dolbec (RETIRED) 2015-03-04 21:29:01 UTC]

Released in portage-2.2.16

[Comment 19 marco 2015-07-20 12:31:03 UTC]

Are these "deleterious effects on the ability of emerge --jobs to parallelize builds" true also for hosts deployed with FEATURES="getbinpkg" or in the case of binary packages is not relevant ?

Moving a private profile to profile-formats = portage-2 profile-set
and migrating from @system to @profile in the packages file speedup the installation of binary packages in parallel where possible  ?

thanks

[Comment 20 Zac Medico 2015-07-20 18:17:28 UTC]

(In reply to marco from comment #19)
> Are these "deleterious effects on the ability of emerge --jobs to
> parallelize builds" true also for hosts deployed with FEATURES="getbinpkg"
> or in the case of binary packages is not relevant ?

Yes. System packages and their dependencies are treated very much the same regardless of whether they are installed a binary packages or built directly from source.

> Moving a private profile to profile-formats = portage-2 profile-set
> and migrating from @system to @profile in the packages file speedup the
> installation of binary packages in parallel where possible  ?

Yes, anything that can be migrated from @system to @profile will increase parallelization.