Summary: | FEATURES='network-sandbox' (on client) prevents distcc compilation | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Kalin KOZHUHAROV <kalin> |
Component: | Core | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | hjckr, kalin |
Priority: | Normal | Keywords: | InVCS |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: | https://bugs.gentoo.org/show_bug.cgi?id=658172 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 611328 | ||
Attachments: |
emerge --info
bin/socks5-server.py: convert address from bytes to str bin/socks5-server.py: convert address from bytes to str |
Description
Kalin KOZHUHAROV
2017-01-03 07:07:40 UTC
We've already got proxy support if your default python interpreter is python3.4 or later: https://gitweb.gentoo.org/proj/portage.git/commit/?id=8fd09dc9a25fb673426340a23794df7f11a44010 Please use the `eselect python` command to check that your default python interpreter is python3.4 or later, and try again. You need to have FEATURES="distcc" enabled (FEATURES="network-sandbox-proxy" also enables the proxy independently). We might be able to add python2.7 support via trollius, but realistically, we can probably rely on people using python3.4 or later. Note that trollius is deprecated: http://trollius.readthedocs.io/deprecated.html I have python-3.4.5 installed and selected, but it does not work for me on stable sys-apps/portage-2.3.0[ipc,xattr]. Not sure what is the requirement for asyncio, any useflag? BTW, virtual/python-asyncio is not installed. Anyway, there is a lot of new (to me) info in this bug, may be start with updating the docs on how is it supposed to work, so I can test it properly. The test for not working condition is still no traffic on `tcpdump -n -i any port 3632` on the client. (In reply to Kalin KOZHUHAROV from comment #3) > Anyway, there is a lot of new (to me) info in this bug, may be start with > updating the docs on how is it supposed to work, so I can test it properly. It seems like it's supposed to "just work". I'm going to have to try it to see if if will work for me or not. Maybe we need to fix some error handling to report some problem that occurs on your system. > The test for not working condition is still no traffic on `tcpdump -n -i any > port 3632` on the client. If it's doing the right thing, then you should be able to see a python process running socks5-server.py. Also, there should be PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables exported in the ebuild environment, and they should be visible in the $T/environment file (/var/tmp/portage/$CATEGORY/$PF/temp/environment). Can you validate whether or not these things behave as I describe? Just to be clear, I'd rather see the proxy code (which I have added) gone and just declare distcc incompatible with network-sandbox. It's a bit buggy (nothing preventing it from working but still) and unmaintained. Plus, it requires Gentoo patching on distcc. If I were doing this today, I'd probably focus on a specific setuid wrapper to escape the sandbox. OK, tested again. Ran `watch -n1 'grep PROXY /var/tmp/portage/*/*/temp/environment'` while emerging a few ebuilds, no PROXY-related environment appeared. Also, no proxy-like processes were started. And no traffic on port 3632/tcp. FYI: # emerge --info |grep ^FEATURES FEATURES="assume-digests binpkg-logs cgroup config-protect-if-modified distcc distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" # eselect python list Available Python interpreters: [1] python2.7 [2] python3.4 * And switching to "-network-sandbox" works fine (for positive control). (In reply to Kalin KOZHUHAROV from comment #6) > OK, tested again. > > Ran `watch -n1 'grep PROXY /var/tmp/portage/*/*/temp/environment'` while > emerging a few ebuilds, no PROXY-related environment appeared. Also, no > proxy-like processes were started. And no traffic on port 3632/tcp. It turns out that the PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables don't show up in the environment file because portage filters them out. I tested locally, and was able to observe distcc working with network-sandbox enabled. I used a line like this in /etc/portage/bashrc to verify that the DISTCC_SOCKS_PROXY variable was set: einfo DISTCC_SOCKS_PROXY=$DISTCC_SOCKS_PROXY > And switching to "-network-sandbox" works fine (for positive control). From the "Network is unreachable" messages that you got with DISTCC_VERBOSE=1, it seems like portage is not even setting up the proxy, and that should only happen if you're using python2.7. Please check `emerge --version` output confirm the python version being used. If it doesn't report Python 3.4.x, make sure that sys-apps/portage is built with PYTHON_TARGETS=python3_4 enabled. Created attachment 459342 [details]
emerge --info
OK, finally progress!
I tested again, caught the proxy process running and confirmed the envvar was set with the bashrc hack
However, still no packets flow to the server... Looking at the logs I found an error that was not clear before, I guess it is the core of my problem:
TypeError: must be str, not bytes
Task exception was never retrieved
future: <Task finished coro=<handle_proxy_conn() done, defined at /usr/lib/portage/python3.4/socks5-server.py:26> exception=TypeError('must be str, not bytes',)>
Traceback (most recent call last):
File "/usr/lib64/python3.4/asyncio/tasks.py", line 240, in _step
result = coro.send(None)
File "/usr/lib/portage/python3.4/socks5-server.py", line 100, in handle_proxy_conn
addr, port)
File "/usr/lib64/python3.4/asyncio/streams.py", line 78, in open_connection
lambda: protocol, host, port, **kwds)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 630, in create_connection
type=socket.SOCK_STREAM, proto=proto, flags=flags)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 574, in getaddrinfo
info = _ipaddr_info(host, port, family, type, proto)
File "/usr/lib64/python3.4/functools.py", line 472, in wrapper
result = user_function(*args, **kwds)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 113, in _ipaddr_info
socket.inet_pton(af, host)
Attaching `emerge --info` FYR.
Created attachment 459394 [details, diff]
bin/socks5-server.py: convert address from bytes to str
I can reproduce the "TypeError: must be str, not bytes" exception with python-3.4.5, and it does not occur with python-3.4.3-r1. Decoding the addr variable from bytes to str solves the problem.
(In reply to Zac Medico from comment #9) > Created attachment 459394 [details, diff] [details, diff] > bin/socks5-server.py: convert address from bytes to str Maybe the patch should assume 'idna' encoding rather than 'utf_8'. I think distcc just passes through whatever encoding exists in the config file. Created attachment 459804 [details, diff] bin/socks5-server.py: convert address from bytes to str Updated patch to use the idna codec. Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/c65af3933bff8c63c2ea9ab97be018d1 https://github.com/gentoo/portage/pull/87 This is in the master branch: https://gitweb.gentoo.org/proj/portage.git/commit/?id=0d32c30b1db848e9eac6fe2e216e6373846467a2 Fixed in portage-2.3.5. Hi, on portage 2.3.55 on 32bit arm which had the network-sandbox feature set and default python interpreter 2.7 this bug bit me so bad today. Hours were wasted trying to understand the change in behaviour and the 'failed to connect' and 'Network is unreachable' which just made no sense... Thanks Kalin for addressing this issue with development and providing workarounds and the bug report. I still depend on python2.7 to be the main interpreter on the system, so switching that globally is not an easy way out. (In reply to Nikolay Kichukov from comment #14) > Hi, > on portage 2.3.55 on 32bit arm which had the network-sandbox feature set and > default python interpreter 2.7 this bug bit me so bad today. Hours were > wasted trying to understand the change in behaviour and the 'failed to > connect' and 'Network is unreachable' which just made no sense... > > Thanks Kalin for addressing this issue with development and providing > workarounds and the bug report. > > I still depend on python2.7 to be the main interpreter on the system, so > switching that globally is not an easy way out. You can solve this by rebuilding sys-apps/portage with USE=python_targets_python2_7 disabled, like this: echo sys-apps/portage -python_targets_python2_7 >> /etc/portage/package.use emerge --oneshot --newuse sys-apps/portage Also see bug 658172. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=035582f0e31c071606635aac9cc4ba4b411612e7 commit 035582f0e31c071606635aac9cc4ba4b411612e7 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-01-14 08:11:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-01-16 07:48:59 +0000 tests: add unit test for portage.util.socks5 (FEATURES=network-sandbox-proxy) Bug: https://bugs.gentoo.org/604474 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/util/test_socks5.py | 211 ++++++++++++++++++++++++++++++++++ lib/portage/util/socks5.py | 48 +++++++- 2 files changed, 256 insertions(+), 3 deletions(-) |