So I this is about something I thought was already adopted across the distros. And I was surprised to learn that Gentoo doesn't implement it. At least, searching the tracker didn't find any discussions. At the PyPA Packaging Summit, back in 2019, a lot of discussions happened. And there were folks from multiple downstreams (I recall Conda, Debian, and Fedora at least). One of the topics was coordination between different installers that attempt to manage the same site-packages folder. And the agreement reached was that the installed metadata (the `.dist-info` directory) would have a file called INSTALLER with a string identifying the tool that installed said package. And the idea was that whenever that piece of metadata doesn't belong to the currently running installer, it would know not to step on the toes of one another. I didn't find it in the summit notes (http://bit.ly/pypa2019), but I've located the PEP where it's documented: https://peps.python.org/pep-0627/#optional-installer-file. Here's what Gentoo has (or rather doesn't have) currently: ``` $ ls /usr/lib/python*/site-packages/*.dist-info/INSTALLER zsh: no matches found: /usr/lib/python*/site-packages/*.dist-info/INSTALLER ``` And here's an example of this bit of metadata in Fedora's dnf/rpm managed location and in a pip-managed virtualenv: ``` $ podman run -it --rm fedora:42 [root@b8c8089219cd /]# dnf install -y python3-psycopg3 [...] [root@b8c8089219cd /]# ls /lib/python3.13/site-packages/psycopg-3.2.1.dist-info/ INSTALLER LICENSE.txt METADATA WHEEL top_level.txt [root@b8c8089219cd /]# cat /lib/python3.13/site-packages/psycopg-3.2.1.dist-info/INSTALLER rpm [root@b8c8089219cd /]# python3 -m venv some-venv [root@b8c8089219cd /]# some-venv/bin/pip install psycopg-binary [...] [root@b8c8089219cd /]# cat some-venv/lib/python3.13/site-packages/psycopg_binary-3.2.6.dist-info/INSTALLER pip ``` I believe it should be doable to automate something like `echo portage > INSTALLER`
(In reply to Sviatoslav @webknjaz Sydorenko #StandWithUkraine from comment #0) > So I this is about something I thought was already adopted across the > distros. And I was surprised to learn that Gentoo doesn't implement it. At > least, searching the tracker didn't find any discussions. > > At the PyPA Packaging Summit, back in 2019, a lot of discussions happened. > And there were folks from multiple downstreams (I recall Conda, Debian, and > Fedora at least). One of the topics was coordination between different > installers that attempt to manage the same site-packages folder. And the > agreement reached was that the installed metadata (the `.dist-info` > directory) would have a file called INSTALLER with a string identifying the > tool that installed said package. And the idea was that whenever that piece > of metadata doesn't belong to the currently running installer, it would know > not to step on the toes of one another. I seem to recall having been told that the pip project had zero intention of implementing the PEP after all, and that therefore creating the file will do nothing and accomplish nothing. The PEP is essentially fully deprecated in the heart and mind of the community by the later PEP introducing the ill-designed and user/distro hostile EXTERNALLY-MANAGED. And since the Python Packaging Authority collectively decided that raising an error and telling people to "don't do that unless you agree that it's your fault for --break-system-packages'ing" has not simply punted on the issue, but *solved* the issue such that no further effort is needed, there's no use case for supporting the INSTALLER file. This killed my interest in advocating for that file's creation, both in Gentoo and in other distribution vendors. > I believe it should be doable to automate something like `echo portage > > INSTALLER` Yes, adding the file would be very easy, if desired. We currently delete the RECORD file in the same place we could add an INSTALLER one. Deleting RECORD is a crude hack, but it is definitely supported by pip, since pip cannot uninstall a package without a RECORD (in addition to other effects that lacking a RECORD has). Crude, but works.
Are there any tools that actually read and use that file somehow? My main concern would be installing lots of identical small files with content that's semi-obvious (since you're not supposed to use any other installer in system site-packages).
I'm curious about the value as well but not *against* it. Very curious as to how you noticed this -was it curiosity/spec compliance checking, or as mgorny asks, is there a tool really relying on this?
Pip implemented this around 2020: https://pip.pypa.io/en/stable/news/#b1-2020-04-21 It would use the value in this file when it can't figure out what to uninstall: ``` $ python -Im venv some-venv $ some-venv/bin/python -Im pip install setuptools Collecting setuptools Using cached setuptools-78.1.0-py3-none-any.whl.metadata (6.6 kB) Using cached setuptools-78.1.0-py3-none-any.whl (1.3 MB) Installing collected packages: setuptools Successfully installed setuptools-78.1.0 $ cat some-venv/lib/python3.12/site-packages/setuptools-78.1.0.dist-info/INSTALLER pip $ rm -f some-venv/lib/python3.12/site-packages/setuptools-78.1.0.dist-info/RECORD $ some-venv/bin/python -Im pip uninstall setuptools Found existing installation: setuptools 78.1.0 error: uninstall-no-record-file × Cannot uninstall setuptools 78.1.0 ╰─> The package's contents are unknown: no RECORD file was found for setuptools. hint: You might be able to recover from this via: pip install --force-reinstall --no-deps setuptools==78.1.0 $ echo emerge | tee some-venv/lib/python3.12/site-packages/setuptools-78.1.0.dist-info/INSTALLER emerge $ some-venv/bin/python -Im pip uninstall setuptools Found existing installation: setuptools 78.1.0 error: uninstall-no-record-file × Cannot uninstall setuptools 78.1.0 ╰─> The package's contents are unknown: no RECORD file was found for setuptools. hint: The package was installed by emerge. You should check if it can uninstall the package. I did check that conda still has the INSTALLER file in its test suite, so does pip. I don't know how conda relies on it exactly. I can imagine there might be other tools looking this file up, though. As another data point, I've also checked that Ubuntu 24.04 has this: ``` root@67fa5d3d5033:/# apt install python3-build^C root@67fa5d3d5033:/# tail /usr/lib/python3/dist-packages/*.dist-info/INSTALLER ==> /usr/lib/python3/dist-packages/build-1.0.3.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/more_itertools-10.2.0.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/packaging-24.0.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/pyproject_hooks-1.0.0.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/typing_extensions-4.10.0.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/wheel-0.42.0.dist-info/INSTALLER <== debian ==> /usr/lib/python3/dist-packages/zipp-1.0.0.dist-info/INSTALLER <== debian ``` There's also this document on the installed metadata mentions that it can be of use when there's no RECORD file: https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-installer-file P.S. As for file duplication, if it's a big problem, maybe having some sort of a symlink/hardlink thing would address it. I don't know if it's typical to have enough packages in there to have visible impact, though. By the way, it looks like some emerge-installed projects do preserve the RECORD file for whatever reason: ``` $ ls -l /usr/lib/python*/site-packages/*.dist-info/RECORD -rw-r--r-- 1 root root 1.9K Jan 1 2016 /usr/lib/python3.12/site-packages/installer-0.7.0.dist-info/RECORD ```
(In reply to Sviatoslav @webknjaz Sydorenko #StandWithUkraine from comment #4) > $ some-venv/bin/python -Im pip uninstall setuptools > Found existing installation: setuptools 78.1.0 > error: uninstall-no-record-file > > × Cannot uninstall setuptools 78.1.0 > ╰─> The package's contents are unknown: no RECORD file was found for > setuptools. > > hint: The package was installed by emerge. You should check if it can > uninstall the package. Ah okay, that is an interesting point. Pip doesn't use the INSTALLER file as a flag to say "no, don't uninstall this package managed by some other package manager", it only uses it when it has already refused to uninstall that package, as a hint for what to use instead. But... given that users cannot actually see the message if EXTERNALLY-MANAGED exists, which on a majority of Gentoo systems it will by default (it does not exist on my system)... ... I still don't see how this is useful. (In reply to Sviatoslav @webknjaz Sydorenko #StandWithUkraine from comment #4) > By the way, it looks like some emerge-installed projects do preserve the > RECORD file for whatever reason: > ``` > $ ls -l /usr/lib/python*/site-packages/*.dist-info/RECORD > -rw-r--r-- 1 root root 1.9K Jan 1 2016 > /usr/lib/python3.12/site-packages/installer-0.7.0.dist-info/RECORD This is the bootstrap problem. installer is not installed using the standard wheel installation rules (that also delete RECORD, and would potentially create the INSTALLER file), because those rules need a preexisting installed version of: - dev-python/gpep517 - dev-python/installer So, it manually implements this. See e.g. the gpep517 ebuild: python_install() { python_domodule gpep517 python_newscript - gpep517 <<-EOF #!${EPREFIX}/usr/bin/python import sys from gpep517.__main__ import main sys.exit(main()) EOF } Here, python_domodule is an enhanced `cp` that knows where the currently selected python has its site-packages directory. We write the script in /usr/bin by hand, as you can see. Since gpep517 isn't a library, we don't install any dist-info for it at all. For installer, we unpack both the sdist and the wheel, use the former to run tests and install, and use the latter to copy over the dist-info by hand: distutils_enable_tests pytest python_compile() { python_domodule src/installer "${WORKDIR}"/*.dist-info }
It's not for me to decide, I just wanted to raise the concern because it confused me and googling didn't reveal the reasoning. FTR, I double-checked with Pradyun privately, and here's what he said: ``` [2:33 PM] webknjaz: Hey, do you know what's the current state of the INSTALLER file support in the installed package metadata? [6:47 PM] pradyunsg: As in? [6:47 PM] pradyunsg: AFAIK, it's generated by pip and uv... and no other semantics are attached to that file? [6:49 PM] webknjaz: I suggested Gentoo to write this file but some folks said that somebody suggested it was no longer a thing and they weren't sure it's used by anything [6:50 PM] pradyunsg: I mean, it's not used by anything AFAIK. [6:50 PM] pradyunsg: But, it's definitely still a thing. [6:50 PM] pradyunsg: I think pip refuses to do self checks if it was installed by something other than pip. ```
