On new system, every "emerge --sync" give me such warning: # emerge --sync >>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD .../usr/lib/python3.7/site-packages/urllib3/connection.py:218: BytesWarning: Comparison between bytes and string if SKIP_HEADER not in values: but this seems to be completed successfully, don't know if this is a BUG? If any info required, please tell me
I haven't run into this myself given different setup but reports from the forums[0] started Nov 12 right after urllib 1.26.1 was added on Nov 11, not that I know if people are using ~arch (not in stable). [0] https://forums.gentoo.org/viewtopic-p-8533240.html
It's related to this change in urllib3 https://github.com/urllib3/urllib3/commit/16b7b332fd1b84c2d465f11d17658c1e83d3f20f The SKIP_HEADER value must be unicode for python3, so it looks like something passed in a bytes value to the HTTPConnection putheader method. The problem could be in urllib3 or the requests library that calls urllib3 (requests is called by gemato).
The BytesWarning appears to be triggered entirely by this patch since it removed a conversion from bytes to unicode: https://github.com/urllib3/urllib3/commit/16b7b332fd1b84c2d465f11d17658c1e83d3f20f diff --git a/src/urllib3/_collections.py b/src/urllib3/_collections.py index 8a94ba5f7..da9857e98 100644 --- a/src/urllib3/_collections.py +++ b/src/urllib3/_collections.py @@ -155,7 +155,7 @@ def __setitem__(self, key, val): def __getitem__(self, key): val = self._container[key.lower()] - return ", ".join([six.ensure_str(v, "ascii") for v in val[1:]]) + return ", ".join(val[1:]) def __delitem__(self, key): del self._container[key.lower()]
You can try -bb instead of -b to make BytesWarning errors: python -bb /usr/bin/emerge --sync If it resuls in traceback, please show this traceback.
Hello, I get the same error according to the command python -bb /usr/bin/emerge --sync File "/usr/lib/python3.7/site-packages/portage/util/_async/AsyncFunction.py", line 39, in _run result = self.target(*(self.args or []), **(self.kwargs or {})) File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 165, in sync taskmaster.run_tasks(tasks, func, status, options=task_opts) File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 65, in run_tasks result = getattr(inst, func)(**kwargs) File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 338, in sync return self.update() File "/usr/lib/python3.7/site-packages/portage/sync/modules/rsync/rsync.py", line 147, in update self._refresh_keys(openpgp_env) File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 268, in _refresh_keys if openpgp_env.refresh_keys_wkd(): File "/usr/lib/python3.7/site-packages/gemato/openpgp.py", line 420, in refresh_keys_wkd resp = requests.get(url, proxies=proxies) File "/usr/lib/python3.7/site-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3.7/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/lib/python3.7/http/client.py", line 1277, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1288, in _send_request self.putrequest(method, url, **skips) File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 214, in putrequest return _HTTPConnection.putrequest(self, method, url, *args, **kwargs) File "/usr/lib/python3.7/http/client.py", line 1176, in putrequest self.putheader('Host', host_enc) File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 218, in putheader if SKIP_HEADER not in values: BytesWarning: Comparison between bytes and string Action: sync for repo: gentoo, returned code = 1
urllib3-1.26.2 is now stable and I'm getting the warning as well (I guess everybody will…)
The same warning on sync: /usr/lib/python3.8/site-packages/urllib3/connection.py:218: BytesWarning: Comparison between bytes and string if SKIP_HEADER not in values:
There are potential security consequences to this bug. Having these warnings on STDERR when syncing can dull the attention of sysadmins monitoring their systems for security problems. All the Gentoo systems I manage use a simple script launched daily by cronie that runs something like : eix-sync -q; glsa-check -clnq affected To avoid any false negative, the cron/script combination relies on STDERR content being a real problem and an email is sent to the admin to deal with it (a sync error will eventually hide security problems). So if anything is put on STDERR an email is sent. I've hold off whole system updates because of this bug as I would be overwhelmed with daily errors for the dozens of systems I manage. The only way to avoid this situation currently would be to update my script to look for and ignore this specific error which will make it more complex (currently it is basically a one-liner), error-prone (I don't trust myself with "grep -v" ...) and maybe short-lived depending on when the fix will come. Although there is a way around the problem, depending on the proportion of people using this kind of basic security monitoring it might be appropriate to raise this bug's Importance.
(In reply to Lionel Bouton from comment #8) > There are potential security consequences to this bug. Having these warnings > on STDERR when syncing can dull the attention of sysadmins monitoring their > systems for security problems. If you can downgrade to dev-python/urllib3-1.25.11 then that should suppress the warning.
*** Bug 762418 has been marked as a duplicate of this bug. ***
(In reply to Zac Medico from comment #9) > (In reply to Lionel Bouton from comment #8) > > There are potential security consequences to this bug. Having these warnings > > on STDERR when syncing can dull the attention of sysadmins monitoring their > > systems for security problems. > > If you can downgrade to dev-python/urllib3-1.25.11 then that should suppress > the warning. Yes indeed. For people looking for a quick fix before the actual bugfix, I'm in the process of deploying this in portage.mask on my systems : # urllib3-1.26.2 outputs warnings on STDERR during emerge --sync # masked to avoid mail pollution when checking for vulnerabilities # 1.25.11 is the last version without these warnings at this time. # Remove when the bug is fixed, see https://bugs.gentoo.org/755083 >dev-python/urllib3-1.25.11 I didn't have any problem downgrading from 1.26.2 or preventing the upgrade with this mask entry.
*** Bug 764296 has been marked as a duplicate of this bug. ***
thanks. after downgrading the error is gone.
are there real security issues with this bug related to WKD key retrievment or anything else? i may ask because i try to install stage3-amd64-nomultilib-20210120T214504Z.tar.xz and run into it. downgrade urlib is no option because it's the first emerge --sync.
No, there are no consequences to this warning.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cef35aa3489a1db4ea56b44cad0beabb5b4c0051 commit cef35aa3489a1db4ea56b44cad0beabb5b4c0051 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2021-01-26 16:23:24 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2021-01-26 18:43:08 +0000 dev-python/urllib3: Backport a fix for BytesWarning Closes: https://bugs.gentoo.org/755083 Signed-off-by: Michał Górny <mgorny@gentoo.org> .../files/urllib3-1.26.2-byteswarning.patch | 25 ++++++++ dev-python/urllib3/urllib3-1.26.2-r2.ebuild | 70 ++++++++++++++++++++++ dev-python/urllib3/urllib3-1.26.2-r3.ebuild | 70 ++++++++++++++++++++++ 3 files changed, 165 insertions(+)