Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 953952 - dev-python/curl-cffi: new package (was: net-misc/yt-dlp does not support the --impersonate switch)
Summary: dev-python/curl-cffi: new package (was: net-misc/yt-dlp does not support the ...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal enhancement
Assignee: Default Assignee for New Packages
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-04-17 08:43 UTC by Peter Fox
Modified: 2025-05-05 02:37 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
dev-python/curl-cffi-0.10.0 ebuild (curl-cffi-0.10.0.ebuild,573 bytes, application/vnd.gentoo.ebuild)
2025-04-17 20:56 UTC, Peter Fox
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Fox 2025-04-17 08:43:11 UTC
yt-dlp is failing to download videos, and one possible solution might be to add --impersonate chrome to the command line arguments (see for example https://github.com/orgs/yt-dlp/projects/6/views/1?pane=issue&itemId=76261638&issue=yt-dlp%7Cyt-dlp%7C10837).

Attempting to use yt-dlp with arguments "--impersonate chrome" results in a backtrace:
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.12/yt-dlp", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/__init__.py", line 1100, in main
    _exit(*variadic(_real_main(argv)))
                    ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/__init__.py", line 998, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 759, in __init__
    raise YoutubeDLError(
yt_dlp.utils.YoutubeDLError: Impersonate target "chrome" is not available. Use --list-impersonate-targets to see available targets. You may be missing dependencies required to support this target.

This feature appears to require the pypi curl_cffi package.

Perhaps add a cffi use flag to yt-dlp to add a dependency on a new gentoo package curl_cffi?
Comment 1 Peter Fox 2025-04-17 08:55:22 UTC
It seems curl_cffi in turn depends on curl-impersonate.
Comment 2 Peter Fox 2025-04-17 08:58:05 UTC
The curl-umpersonate fork used by curl_cffi is at https://github.com/lwthiker/curl-impersonate.
Comment 3 Ionen Wolkens gentoo-dev 2025-04-17 09:00:39 UTC
Had a peek at it before, and the curl-impersonate package is rather messy and I'd rather not handle it myself just because I maintain yt-dlp.

If it's ever added (by another developer) I could add a optfeature to mention support (nothing else to do for the yt-dlp side, the dependency just needs to exist at runtime).

Note that it does exist in overlays, e.g. from
https://github.com/Tatsh/tatsh-overlay/tree/master/dev-python/curl-cffi
Comment 4 Ionen Wolkens gentoo-dev 2025-04-17 09:03:20 UTC
(In reply to Ionen Wolkens from comment #3)
> Had a peek at it before, and the curl-impersonate package is rather messy
> and I'd rather not handle it myself just because I maintain yt-dlp.
(could potentially change that stance if I end up needing that myself, but currently do not)
Comment 5 Peter Fox 2025-04-17 09:07:14 UTC
A worrying quote from
https://curl-cffi.readthedocs.io/en/v0.5.9/compile/:

"This build process is very hacky now, but it works for most common systems."

There doesn't seem to be a newer version of this page.
Comment 6 Peter Fox 2025-04-17 20:56:48 UTC
Created attachment 925173 [details]
dev-python/curl-cffi-0.10.0 ebuild

Bumped ebuild from the tatsh-overlay referred to in an earlier ebuild.

yt-dlp-9999 requires curl-cffi 0.5.10 or 0.10.x (see https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/networking/_curlcffi.py).

I also installed net-misc/curl-impersonate-0.9.5 from the tatsh-overlay.

With these yt-dlp appears to be happy:

$ yt-dlp --list-impersonate-targets
[info] Available impersonate targets
Client        OS           Source
------------------------------------
Safari-17.2   Ios-17.2     curl_cffi
Safari-18.0   Ios-18.0     curl_cffi
Chrome-99     Android-12   curl_cffi
Chrome-131    Android-14   curl_cffi
Edge-99       Windows-10   curl_cffi
Edge-101      Windows-10   curl_cffi
Firefox-133   Macos-14     curl_cffi
Firefox-135   Macos-14     curl_cffi
Safari-15.3   Macos-14     curl_cffi
Safari-15.5   Macos-14     curl_cffi
Safari-17.0   Macos-14     curl_cffi
Safari-18.0   Macos-15     curl_cffi
Chrome-99     Windows-10   curl_cffi
Chrome-100    Windows-10   curl_cffi
Chrome-101    Windows-10   curl_cffi
Chrome-104    Windows-10   curl_cffi
Chrome-107    Windows-10   curl_cffi
Chrome-110    Windows-10   curl_cffi
Chrome-116    Windows-10   curl_cffi
Chrome-119    Macos-14     curl_cffi
Chrome-120    Macos-14     curl_cffi
Chrome-123    Macos-14     curl_cffi
Chrome-124    Macos-14     curl_cffi
Chrome-131    Macos-14     curl_cffi
Chrome-133    Macos-15     curl_cffi

But it doesn't work around my original download issue :-(
Comment 7 Paweł Metelski 2025-05-05 02:37:49 UTC
Fetching the same movie with portage's yt-dlp-2025.03.31 fails with an error:
```
An extractor error has occurred. (caused by KeyError('config_url')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```
while downloading yt-dlp with `pip install "yt-dlp[default,curl-cffi]"` (2025.4.30) downloads it without issues (however slowly).