Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 920330 - sys-apps/portage: does not install .dist-info anymore, breaking Python-level dependencies
Summary: sys-apps/portage: does not install .dist-info anymore, breaking Python-level ...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 925481
Blocks:
  Show dependency tree
 
Reported: 2023-12-19 16:35 UTC by Michał Górny
Modified: 2024-04-28 00:32 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2023-12-19 16:35:12 UTC
$ pip check | grep -v certifi
pfl 3.3.1 requires portage, which is not installed.
Comment 1 Eli Schwartz 2023-12-19 16:57:37 UTC
This is impossible to do using the python packaging ecosystem, since PEP 517 does not acknowledge the possibility of installing files to /etc and considers installing files to /usr but not "specifically inside the site-packages directory" to be under-specified, deprecated, unreliable across operating systems, terminally broken in virtualenvs, and "best avoided because will pressure you into considering it a violation of sound design principles".

Accordingly, the only winning move is not to play. Don't try to be a python ecosystem software, instead be system software that happens to be written in python. You lose dist-info that almost no one cares about (even pip doesn't care about it! It happily ignores dependency conflicts) and in return you get the ability to deploy working software that isn't limited by the idealistic notion that the only software is:

- libraries
- occasionally, CLI entrypoints that run a library function
- Django websites and other code that runs from a git checkout by running a `./foo.py` script inside a poetry managed virtualenv
Comment 2 Eli Schwartz 2023-12-19 17:00:55 UTC
The ebuild installs portage with meson.eclass instead of DISTUTILS_USE_PEP517.

In theory one could do both and install twice. The second time just to get the dist-info.

In the long term the goal is that meson will natively install dist-info without requiring to build pointless wheels (though you could if you wanted to).
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2023-12-19 17:47:02 UTC
CC-ing billie@ as maintainer of app-portage/pfl.
Comment 4 James Le Cuirot gentoo-dev 2023-12-19 21:17:53 UTC
IIRC the installation layout is a bit different with meson-python, so I don't think building a wheel to get these files would work.

I think this should probably be Meson's job, even if it only covers the files installed under site-packages.

As Eli says, do we even need this? You can still do "import portage" just fine.
Comment 5 Daniel Pielmeier gentoo-dev 2023-12-20 17:26:51 UTC
(In reply to Michał Górny from comment #3)
> CC-ing billie@ as maintainer of app-portage/pfl.

I am a bit lost here. Is this a pfl or portage issue?

Also dist-info is installed here. Is it supposed to be a single file named .dist-info? Here it is a directory named dist-info containing multiple files e.g. METADATA

Recently pfl switched from setup.py to pyproject.toml. I guess this is unrelated but some time afterwards the dependencies [1] have been specified in the toml file. So this message pops up for pfl-3.3.1 (here the specified dependencies are included in dist-info/METADATA) but not for pfl-3.2.1 which doesn't include the dependencies.

This also occurs with app-portage/elogv and media-sound/puddletag which also specify the dependencies in setup.py respectively pyproject.toml

I don't know if this is helpful because I guess you are aware of this.

[1] https://github.com/portagefilelist/client/commit/2fc588e1416238e2f179f49240e9c9e8da314dc5


(In reply to Eli Schwartz from comment #2)
> The ebuild installs portage with meson.eclass instead of
> DISTUTILS_USE_PEP517.
> 
> In theory one could do both and install twice. The second time just to get
> the dist-info.
> 
> In the long term the goal is that meson will natively install dist-info
> without requiring to build pointless wheels (though you could if you wanted
> to).

I guess you are talking about a different ebuild. pfl (and also puddletag) use DISTUTILS_USE_PEP517.
Comment 6 Alfred Wingate 2023-12-20 17:33:43 UTC
Its a portage issue, its just apparent now that portage is listed as a dependency in pfl metadata. It also shows up from elogv where I also added portage as a dependency in the metadata.


Having portage as a dependency made testing them easier in a venv as it worked as expected if EROOT  / EPREFIX was correctly set.
Comment 7 James Le Cuirot gentoo-dev 2023-12-21 21:22:27 UTC
Believe it or not, this was enough to satisfy pip check:

/usr/lib/python3.12/site-packages/portage-3.0.58.dist-info/METADATA

> Metadata-Version: 2.1
> Name: portage
> Version: 3.0.58

So we could easily just write that out as part of the regular Portage build with some of the other fields for good measure.

What other tools use these files?
Comment 8 Eli Schwartz 2023-12-21 21:27:26 UTC
(In reply to James Le Cuirot from comment #7)
> What other tools use these files?

sudo pip install thing-that-breaks-portage-vdb
Comment 9 James Le Cuirot gentoo-dev 2023-12-21 21:34:15 UTC
(In reply to Eli Schwartz from comment #8)
> (In reply to James Le Cuirot from comment #7)
> > What other tools use these files?
> 
> sudo pip install thing-that-breaks-portage-vdb

Let me put that another way. Is there anything I should test to ensure that writing this minimal amount of metadata doesn't cause breakage?
Comment 10 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2023-12-22 07:05:09 UTC
I'd really prefer if we did things properly and had Portage installed via a proper build system.  Even if it means splitting the Python and the non-Python part.
Comment 11 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-02-27 10:05:19 UTC
Ping.
Comment 12 Zac Medico gentoo-dev 2024-02-27 17:01:20 UTC
(In reply to Michał Górny from comment #10)
> I'd really prefer if we did things properly and had Portage installed via a
> proper build system.  Even if it means splitting the Python and the
> non-Python part.

We could extend the portage.installation.TYPES Enum to include a split type that is a hybrid of the MODULE and SYSTEM types:

TYPES = Enum(
    "InstallationType",
    [
        "SOURCE",  # Portage is not installed, but running from its source tree.
        "MODULE",  # Portage is installed solely as a Python module.
        "SYSTEM",  # Portage is fully installed to the system, possibly prefixed.
    ],
)

For the split type, the portage.const module will have to refer to paths that are installed separately from the wheel.
Comment 13 Zac Medico gentoo-dev 2024-03-06 07:29:37 UTC
I thought it might be nice to combine distutils-r1 and meson eclasses, and use meson to generate a portage/installation.py for each python implementation, overwriting the one from the wheel.

However, if you try to combine them like this then you end up with installation.py from distutils-r1 because both src_compile functions are using the same multibuild build dirs:

src_compile() {
	distutils-r1_src_compile
	python_foreach_impl meson_src_compile
}
Comment 14 Zac Medico gentoo-dev 2024-03-07 04:21:37 UTC
Combining distutils-r1 and python-r1 eclasses works if we copy ${S} to ${S}-meson in src_prepare and then override BUILD_DIR for the python-r1 python_foreach_impl invocations like this:

src_compile() {
	distutils-r1_src_compile
	BUILD_DIR="${S}-meson" python_foreach_impl meson_src_compile
}
Comment 15 Larry the Git Cow gentoo-dev 2024-03-08 16:11:50 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=32e4a588cfd9548a28c74476d586a6b66b15bace

commit 32e4a588cfd9548a28c74476d586a6b66b15bace
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-03-08 05:04:10 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-03-08 05:04:10 +0000

    meson.build: Install dist-info METADATA
    
    Install dist-info METADATA for pip to resolve dependencies:
    
        # pip freeze | grep portage
        portage==3.0.63
    
    Suggested-by: Eli Schwartz <eschwartz93@gmail.com>
    Bug: https://bugs.gentoo.org/920330
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 cnf/METADATA |  3 +++
 meson.build  | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
Comment 16 Larry the Git Cow gentoo-dev 2024-04-28 00:32:42 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f1e53b9368c215451e3af78b557d8b86a2db9677

commit f1e53b9368c215451e3af78b557d8b86a2db9677
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-04-28 00:31:29 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-04-28 00:31:29 +0000

    sys-apps/portage: add 3.0.64
    
    Closes: https://bugs.gentoo.org/925481
    Closes: https://bugs.gentoo.org/675748
    Closes: https://bugs.gentoo.org/712222
    Closes: https://bugs.gentoo.org/920330
    Closes: https://bugs.gentoo.org/925213
    Closes: https://bugs.gentoo.org/925863
    Closes: https://bugs.gentoo.org/925888
    Closes: https://bugs.gentoo.org/925907
    Closes: https://bugs.gentoo.org/926048
    Closes: https://bugs.gentoo.org/926221
    Closes: https://bugs.gentoo.org/927059
    Closes: https://bugs.gentoo.org/927632
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.64.ebuild | 242 +++++++++++++++++++++++++++++++++
 2 files changed, 243 insertions(+)