Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 717666 - dev-python/pipenv: bundles humongous number of packages
Summary: dev-python/pipenv: bundles humongous number of packages
Status: IN_PROGRESS
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: OzTiram
URL:
Whiteboard:
Keywords: PullRequest
Depends on: 830900 830801
Blocks: bundled-libs
  Show dependency tree
 
Reported: 2020-04-16 08:31 UTC by Michał Górny
Modified: 2022-01-10 08:39 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-04-16 08:31:03 UTC
# ls /usr/lib/python3.7/site-packages/pipenv/vendor/
appdirs.LICENSE.txt      click                 dotenv         parse.LICENSE       ptyprocess           scandir.py          vendor_pip.txt
appdirs.py               click_completion      first.LICENSE  parse.py            __pycache__          semver.LICENSE.txt  vendor.txt
attr                     click_didyoumean      first.py       passa               pyparsing.LICENSE    semver.py           vistir
backports                colorama              idna           pathlib2            pyparsing.py         shellingham         yarg
blindspin                cursor                __init__.py    pexpect             pythonfinder         shutilwhich         yaspin
cached-property.LICENSE  delegator.py          iso8601        pipdeptree.LICENSE  README.md            six.LICENSE
cached_property.py       delegator.py.LICENSE  jinja2         pipdeptree.py       requests             six.py
cerberus                 distlib               Makefile       pipreqs             requirementslib      toml
certifi                  docopt.LICENSE-MIT    markupsafe     pip_shims           resolvelib           tomlkit
chardet                  docopt.py             packaging      plette              scandir.LICENSE.txt  urllib3

# ls /usr/lib/python3.7/site-packages/pipenv/patched/
crayons.LICENSE  crayons.py  __init__.py  notpip  patched.txt  pipfile  piptools  __pycache__  README.md  safety  safety.zip
Comment 1 Ralph Seichter 2020-04-18 20:47:58 UTC
I may be interested in becoming the new maintainer for this package, so I had a look at open bugs. This one made me wonder what you consider a "fix"? https://github.com/pypa/pipenv/blob/master/setup.py lists quite a number of dependencies, plus upper-bound version limitations. It looks like changes would need to be made upstream, not in a Gentoo ebuild.
Comment 2 Sam James archtester gentoo-dev Security 2021-03-03 17:33:56 UTC
(In reply to Ralph Seichter from comment #1)
> I may be interested in becoming the new maintainer for this package, so I
> had a look at open bugs. This one made me wonder what you consider a "fix"?
> https://github.com/pypa/pipenv/blob/master/setup.py lists quite a number of
> dependencies, plus upper-bound version limitations. It looks like changes
> would need to be made upstream, not in a Gentoo ebuild.

The fix would be unbundling these and using system versions.

Often, these upper-bound version limitations are conservative or even wrong. But yes, it's possible some porting work would be needed to fix some and upstream those fixes.

(I added you to CC because I'm replying now and it looks like you weren't in it before).
Comment 3 OzTiram 2021-07-23 22:17:08 UTC
This is in work. See related PR.
Comment 4 Larry the Git Cow gentoo-dev 2021-08-03 07:09:55 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8e0be56464cc537990c3ae58a9f89cca081af059

commit 8e0be56464cc537990c3ae58a9f89cca081af059
Author:     Oz N Tiram <oz.tiram@gmail.com>
AuthorDate: 2021-07-23 22:01:14 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2021-08-03 07:09:49 +0000

    dev-python/pipenv: remove vendored jinja2 and attr.
    
    Since pipenv bundles tons of packages, these will be
    removed in a slow pace. If no bugs are reported, the
    removal of bundled packages and addition of dependencies
    can continue.
    
    Bug: https://bugs.gentoo.org/717666
    
    Signed-off-by: Oz N Tiram <oz.tiram@gmail.com>
    Closes: https://github.com/gentoo/gentoo/pull/21828
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 ...nv-2021-5-29-r1-remove-attr-vendor-import.patch | 95 ++++++++++++++++++++++
 dev-python/pipenv/pipenv-2021.5.29-r1.ebuild       | 54 ++++++++++++
 2 files changed, 149 insertions(+)
Comment 5 Grzegorz Herman 2021-08-09 07:24:32 UTC
Unbundling attr seems to have broken my installation of pipenv:

$ pipenv --version
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.9/pipenv", line 33, in <module>
    sys.exit(load_entry_point('pipenv==2021.5.29', 'console_scripts', 'pipenv')())
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 1044, in parse_args
    parser = self.make_parser(ctx)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 965, in make_parser
    for param in self.get_params(ctx):
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 912, in get_params
    help_option = self.get_help_option(ctx)
  File "/usr/lib/python3.9/site-packages/pipenv/cli/options.py", line 27, in get_help_option
    from ..core import format_help
  File "/usr/lib/python3.9/site-packages/pipenv/core.py", line 33, in <module>
    from .project import Project
  File "/usr/lib/python3.9/site-packages/pipenv/project.py", line 30, in <module>
    from .vendor.requirementslib.models.utils import get_default_pyproject_backend
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/requirementslib/__init__.py", line 9, in <module>
    from .models.lockfile import Lockfile
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 8, in <module>
    import attr
ModuleNotFoundError: No module named 'attr'
Comment 6 OzTiram 2021-08-09 08:00:39 UTC
Hi Herman,

Thank you for the report.
Yes, you are right. This was dumb. 

Removing attr and not adding as explicit dependency breaks it.

So, the next step is:

add the following explicit dependencies:

 1. attr
 2. jinja2
 3. yaml

Completely remove yaml.
 
For each vendor package removed, explicitly add gentoo ebuild dependency.

I will attend to it asap.
Comment 7 OzTiram 2021-08-10 11:50:25 UTC
In order for me to close this bug, I will need to add a few packages to the main gentoo repository, skipping guru.

I will also search if some of those already exist in guru, and if so, I can ask here to promote them to the main repo.


@juppis will you approve such process?
Comment 8 Joonas Niilola gentoo-dev 2021-08-10 12:47:20 UTC
(In reply to OzTiram from comment #7)
> In order for me to close this bug, I will need to add a few packages to the
> main gentoo repository, skipping guru.
> 
> I will also search if some of those already exist in guru, and if so, I can
> ask here to promote them to the main repo.
> 
> 
> @juppis will you approve such process?

Please go for it, unless we're talking about hundreds of packages here.
Comment 9 Larry the Git Cow gentoo-dev 2021-08-18 13:33:27 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6f583f0ebf698afcd5cc227f00aae83837f4047c

commit 6f583f0ebf698afcd5cc227f00aae83837f4047c
Author:     Oz N Tiram <oz.tiram@gmail.com>
AuthorDate: 2021-08-10 08:34:49 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2021-08-18 13:33:16 +0000

    dev-python/pipenv: bump rev-2
    
     * Disable py3.7
     * Fix attr->attrs
    
    Signed-off-by: Oz N Tiram <oz.tiram@gmail.com>
    Bug: https://bugs.gentoo.org/show_bug.cgi?id=717666
    Closes: https://github.com/gentoo/gentoo/pull/21925
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 ...v-2021-5-29-r2-remove-attr-vendor-import.patch} |  0
 ...21-5-29-r2-remove-colorama-vendor-import.patch} |  0
 dev-python/pipenv/pipenv-2021.5.29-r2.ebuild       | 61 ++++++++++++++++++++++
 3 files changed, 61 insertions(+)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7d2579a842be44fd0d4ed5fe50c37820d2344348

commit 7d2579a842be44fd0d4ed5fe50c37820d2344348
Author:     Oz N Tiram <oz.tiram@gmail.com>
AuthorDate: 2021-08-09 11:40:29 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2021-08-18 13:33:15 +0000

    dev-python/pipenv: fixed vendored packages
    
     * Fix missing deps on jinja and attr
     * Remove vendored colorama, add dep on system colorama
     * Enable Python-3.10
    
    Signed-off-by: Oz N Tiram <oz.tiram@gmail.com>
    Bug: https://bugs.gentoo.org/show_bug.cgi?id=717666
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 ...nv-2021-5-29-r1-remove-colorama-vendor-import.patch | 18 ++++++++++++++++++
 dev-python/pipenv/pipenv-2021.5.29-r1.ebuild           | 11 +++++++++--
 2 files changed, 27 insertions(+), 2 deletions(-)
Comment 10 OzTiram 2021-11-11 13:06:11 UTC
Status update:

The latest version[1] of pipenv removes 5 bundeled packages (out of 42 in total).
I hope to remove even larger amount of packages in the next version.





[1]: https://github.com/gentoo/gentoo/pull/22900#issuecomment-966285556