Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 755083 - >=dev-python/urllib3-1.26.0: emerge --sync warning: urllib3/connection.py:218: BytesWarning: Comparison between bytes and string
Summary: >=dev-python/urllib3-1.26.0: emerge --sync warning: urllib3/connection.py:218...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal with 3 votes (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
: 762418 764296 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-11-17 16:39 UTC by Leonid Kopylov
Modified: 2021-01-26 18:43 UTC (History)
29 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Leonid Kopylov 2020-11-17 16:39:13 UTC
On new system, every "emerge --sync" give me such warning:

# emerge --sync
>>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'...
 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys via WKD .../usr/lib/python3.7/site-packages/urllib3/connection.py:218: BytesWarning: Comparison between bytes and string
  if SKIP_HEADER not in values:



but this seems to be completed successfully, don't know if this is a BUG? If any info required, please tell me
Comment 1 Ionen Wolkens gentoo-dev 2020-11-19 17:08:07 UTC
I haven't run into this myself given different setup but reports from the forums[0] started Nov 12 right after urllib 1.26.1 was added on Nov 11, not that I know if people are using ~arch (not in stable).

[0] https://forums.gentoo.org/viewtopic-p-8533240.html
Comment 2 Zac Medico gentoo-dev 2020-11-19 19:30:42 UTC
It's related to this change in urllib3

https://github.com/urllib3/urllib3/commit/16b7b332fd1b84c2d465f11d17658c1e83d3f20f

The SKIP_HEADER value must be unicode for python3, so it looks like something passed in a bytes value to the HTTPConnection putheader method. The problem could be in urllib3 or the requests library that calls urllib3 (requests is called by gemato).
Comment 3 Zac Medico gentoo-dev 2020-11-19 19:52:23 UTC
The BytesWarning appears to be triggered entirely by this patch since it removed a conversion from bytes to unicode:

https://github.com/urllib3/urllib3/commit/16b7b332fd1b84c2d465f11d17658c1e83d3f20f

diff --git a/src/urllib3/_collections.py b/src/urllib3/_collections.py
index 8a94ba5f7..da9857e98 100644
--- a/src/urllib3/_collections.py
+++ b/src/urllib3/_collections.py
@@ -155,7 +155,7 @@ def __setitem__(self, key, val):
 
     def __getitem__(self, key):
         val = self._container[key.lower()]
-        return ", ".join([six.ensure_str(v, "ascii") for v in val[1:]])
+        return ", ".join(val[1:])
 
     def __delitem__(self, key):
         del self._container[key.lower()]
Comment 4 Arfrever Frehtes Taifersar Arahesis 2020-11-20 20:36:22 UTC
You can try -bb instead of -b to make BytesWarning errors:

python -bb /usr/bin/emerge --sync

If it resuls in traceback, please show this traceback.
Comment 5 Juan Pedro 2020-11-22 11:17:59 UTC
Hello, I get the same error

according to the command python -bb /usr/bin/emerge --sync

  File "/usr/lib/python3.7/site-packages/portage/util/_async/AsyncFunction.py", line 39, in _run
    result = self.target(*(self.args or []), **(self.kwargs or {}))
  File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 165, in sync
    taskmaster.run_tasks(tasks, func, status, options=task_opts)
  File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 65, in run_tasks
    result = getattr(inst, func)(**kwargs)
  File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 338, in sync
    return self.update()
  File "/usr/lib/python3.7/site-packages/portage/sync/modules/rsync/rsync.py", line 147, in update
    self._refresh_keys(openpgp_env)
  File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 268, in _refresh_keys
    if openpgp_env.refresh_keys_wkd():
  File "/usr/lib/python3.7/site-packages/gemato/openpgp.py", line 420, in refresh_keys_wkd
    resp = requests.get(url, proxies=proxies)
  File "/usr/lib/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1288, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 214, in putrequest
    return _HTTPConnection.putrequest(self, method, url, *args, **kwargs)
  File "/usr/lib/python3.7/http/client.py", line 1176, in putrequest
    self.putheader('Host', host_enc)
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 218, in putheader
    if SKIP_HEADER not in values:
BytesWarning: Comparison between bytes and string

Action: sync for repo: gentoo, returned code = 1
Comment 6 Erik Quaeghebeur 2020-12-24 20:34:49 UTC
urllib3-1.26.2 is now stable and I'm getting the warning as well (I guess everybody will…)
Comment 7 Drew 2020-12-28 08:27:26 UTC
The same warning on sync:

/usr/lib/python3.8/site-packages/urllib3/connection.py:218: BytesWarning: Comparison between bytes and string
  if SKIP_HEADER not in values:
Comment 8 Lionel Bouton 2020-12-28 13:51:37 UTC
There are potential security consequences to this bug. Having these warnings on STDERR when syncing can dull the attention of sysadmins monitoring their systems for security problems.

All the Gentoo systems I manage use a simple script launched daily by cronie that runs something like :
eix-sync -q; glsa-check -clnq affected

To avoid any false negative, the cron/script combination relies on STDERR content being a real problem and an email is sent to the admin to deal with it (a sync error will eventually hide security problems).

So if anything is put on STDERR  an email is sent. I've hold off whole system updates because of this bug as I would be overwhelmed with daily errors for the dozens of systems I manage.
The only way to avoid this situation currently would be to update my script to look for and ignore this specific error which will make it more complex (currently it is basically a one-liner), error-prone (I don't trust myself with "grep -v" ...) and maybe short-lived depending on when the fix will come.

Although there is a way around the problem, depending on the proportion of people using this kind of basic security monitoring it might be appropriate to raise this bug's Importance.
Comment 9 Zac Medico gentoo-dev 2020-12-28 23:11:43 UTC
(In reply to Lionel Bouton from comment #8)
> There are potential security consequences to this bug. Having these warnings
> on STDERR when syncing can dull the attention of sysadmins monitoring their
> systems for security problems.

If you can downgrade to dev-python/urllib3-1.25.11 then that should suppress the warning.
Comment 10 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2020-12-29 12:05:28 UTC
*** Bug 762418 has been marked as a duplicate of this bug. ***
Comment 11 Lionel Bouton 2020-12-29 13:39:22 UTC
(In reply to Zac Medico from comment #9)
> (In reply to Lionel Bouton from comment #8)
> > There are potential security consequences to this bug. Having these warnings
> > on STDERR when syncing can dull the attention of sysadmins monitoring their
> > systems for security problems.
> 
> If you can downgrade to dev-python/urllib3-1.25.11 then that should suppress
> the warning.

Yes indeed. For people looking for a quick fix before the actual bugfix, I'm in the process of deploying this in portage.mask on my systems :

# urllib3-1.26.2 outputs warnings on STDERR during emerge --sync
# masked to avoid mail pollution when checking for vulnerabilities
# 1.25.11 is the last version without these warnings at this time.
# Remove when the bug is fixed, see https://bugs.gentoo.org/755083
>dev-python/urllib3-1.25.11

I didn't have any problem downgrading from 1.26.2 or preventing the upgrade with this mask entry.
Comment 12 Brian Evans (RETIRED) gentoo-dev 2021-01-07 13:49:40 UTC
*** Bug 764296 has been marked as a duplicate of this bug. ***
Comment 13 Markus Giese 2021-01-13 09:55:53 UTC
thanks. after downgrading the error is gone.
Comment 14 salmander 2021-01-23 21:13:57 UTC
are there real security issues with this bug related to WKD key retrievment or anything else?

i may ask because i try to install stage3-amd64-nomultilib-20210120T214504Z.tar.xz and run into it.

downgrade urlib is no option because it's the first emerge --sync.
Comment 15 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-01-24 09:38:44 UTC
No, there are no consequences to this warning.
Comment 16 Larry the Git Cow gentoo-dev 2021-01-26 18:43:13 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cef35aa3489a1db4ea56b44cad0beabb5b4c0051

commit cef35aa3489a1db4ea56b44cad0beabb5b4c0051
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2021-01-26 16:23:24 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2021-01-26 18:43:08 +0000

    dev-python/urllib3: Backport a fix for BytesWarning
    
    Closes: https://bugs.gentoo.org/755083
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 .../files/urllib3-1.26.2-byteswarning.patch        | 25 ++++++++
 dev-python/urllib3/urllib3-1.26.2-r2.ebuild        | 70 ++++++++++++++++++++++
 dev-python/urllib3/urllib3-1.26.2-r3.ebuild        | 70 ++++++++++++++++++++++
 3 files changed, 165 insertions(+)