Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 821511 - Vendor/bundle Portage's Python dependencies to ease upgrades (static portage/emerge)
Summary: Vendor/bundle Portage's Python dependencies to ease upgrades (static portage/...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-11-03 17:59 UTC by Sam James
Modified: 2023-03-03 03:20 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 17:59:35 UTC
Upgrading Portage itself is rather tricky when e.g. the default value of PYTHON_COMPAT in the profiles has changed (from say, Python 3.8 -> Python 3.9).

We may want to consider vendoring/bundling Portage's Python dependencies to decouple it from the Python eclasses and make it far easier to just run 'emerge -v1 portage' if hitting e.g. unsupported (new) EAPI issues.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-11-03 18:52:51 UTC
All Portage's Python dependencies are optional.  Not that there's many of them.
Comment 2 Arfrever Frehtes Taifersar Arahesis 2021-11-03 22:28:35 UTC
Except especially dev-python/setuptools (only build time), but bundling dev-python/setuptools seems like very bad idea.


setup.py of Portage currently still has fallback code:
> try:
>     from setuptools.core import setup, Command, Extension
>     ...
> except ImportError:
>     from distutils.core import setup, Command, Extension
>     ...

However distutils fallback could no longer work with Python 3.12 which will no longer install distutils modules. (https://www.python.org/dev/peps/pep-0632/)
Comment 3 Martin Gysel (bearsh) 2021-11-04 07:12:27 UTC
I've successfully used dev-python/nuitka or pyinstaller in the past in some projects. both are able to generate a single binary with all deps included.

In case of nuitka, it may even speedup portage due to converting some python code to c.

portage may then check if a newer version of that static-portage is available and offer that to the user and simple pass all arguments through to the static version.
Comment 4 Arfrever Frehtes Taifersar Arahesis 2021-11-04 11:58:38 UTC
(In reply to Sam James from comment #0)
> Upgrading Portage itself is rather tricky when e.g. the default value of
> PYTHON_COMPAT in the profiles has changed (from say, Python 3.8 -> Python
> 3.9).

Instead of bundling, another possibility is to make profiles enable 2 python_targets_* USE flags for dependencies of Portage for some time.

E.g. when in profiles/base/make.defaults PYTHON_TARGETS="python3_9" is changed to PYTHON_TARGETS="python3_10", then profiles/base/package.use would contain:

app-portage/gemato python_targets_python3_9
dev-python/PySocks python_targets_python3_9
dev-python/certifi python_targets_python3_9
dev-python/charset_normalizer python_targets_python3_9
dev-python/idna python_targets_python3_9
dev-python/requests python_targets_python3_9
dev-python/setuptools python_targets_python3_9
dev-python/setuptools_scm python_targets_python3_9
dev-python/packaging python_targets_python3_9
dev-python/pyparsing python_targets_python3_9
dev-python/tomli python_targets_python3_9
dev-python/urllib3 python_targets_python3_9
sys-libs/libselinux python_targets_python3_9

# Below packages are needed only by dev-python/urllib3[brotli]:
? dev-python/brotlicffi python_targets_python3_9
? virtual/python-cffi python_targets_python3_9
? dev-python/cffi python_targets_python3_9
? dev-python/pycparser python_targets_python3_9
? dev-python/ply python_targets_python3_9

(This list needs to be re-checked in future.)

Such entries might remain for 1 year, or until change of PYTHON_TARGETS="python3_10" to PYTHON_TARGETS="python3_11" in profiles/base/make.defaults.

These users, who after migration do not set "*/* PYTHON_TARGETS: -python3_9" in /etc/portage/package.use, would keep 2 slots of Python installed, but it is not unprecedented, since 2.7 and 3.* co-existed for many years.


(It is probably too late to add python_targets_python3_8 entries.)