Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 604474

Summary: FEATURES='network-sandbox' (on client) prevents distcc compilation
Product: Portage Development Reporter: Kalin KOZHUHAROV <kalin>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: hjckr, kalin
Priority: Normal Keywords: InVCS
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=658172
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 611328    
Attachments: emerge --info
bin/socks5-server.py: convert address from bytes to str
bin/socks5-server.py: convert address from bytes to str

Description Kalin KOZHUHAROV 2017-01-03 07:07:40 UTC
FEATURES='network-sandbox' (on client) prevents distcc compilation and took me long time to resolve. If it was intended to do so, please update documentation (man-page). I have edited the wiki: https://wiki.gentoo.org/wiki/Distcc#With_Portage

I have been trying to set up a new pair of systems with distcc but had issues, so I wrote a forum post: https://forums.gentoo.org/viewtopic-t-1057054-highlight-.html


Reproducible: Always
Comment 1 Zac Medico gentoo-dev 2017-01-06 16:33:35 UTC
We've already got proxy support if your default python interpreter is python3.4 or later:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8fd09dc9a25fb673426340a23794df7f11a44010

Please use the `eselect python` command to check that your default python interpreter is python3.4 or later, and try again. You need to have FEATURES="distcc" enabled (FEATURES="network-sandbox-proxy" also enables the proxy independently).
Comment 2 Zac Medico gentoo-dev 2017-01-06 16:51:13 UTC
We might be able to add python2.7 support via trollius, but realistically, we can probably rely on people using python3.4 or later. Note that trollius is deprecated:

http://trollius.readthedocs.io/deprecated.html
Comment 3 Kalin KOZHUHAROV 2017-01-06 21:20:34 UTC
I have python-3.4.5 installed and selected, but it does not work for me on stable sys-apps/portage-2.3.0[ipc,xattr].

Not sure what is the requirement for asyncio, any useflag? BTW, virtual/python-asyncio is not installed.

Anyway, there is a lot of new (to me) info in this bug, may be start with updating the docs on how is it supposed to work, so I can test it properly.

The test for not working condition is still no traffic on `tcpdump -n -i any port 3632` on the client.
Comment 4 Zac Medico gentoo-dev 2017-01-06 22:05:46 UTC
(In reply to Kalin KOZHUHAROV from comment #3)
> Anyway, there is a lot of new (to me) info in this bug, may be start with
> updating the docs on how is it supposed to work, so I can test it properly.

It seems like it's supposed to "just work". I'm going to have to try it to see if if will work for me or not. Maybe we need to fix some error handling to report some problem that occurs on your system.

> The test for not working condition is still no traffic on `tcpdump -n -i any
> port 3632` on the client.

If it's doing the right thing, then you should be able to see a python process running socks5-server.py. Also, there should be PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables exported in the ebuild environment, and they should be visible in the $T/environment file (/var/tmp/portage/$CATEGORY/$PF/temp/environment). Can you validate whether or not these things behave as I describe?
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-01-06 22:18:11 UTC
Just to be clear, I'd rather see the proxy code (which I have added) gone and just declare distcc incompatible with network-sandbox. It's a bit buggy (nothing preventing it from working but still) and unmaintained. Plus, it requires Gentoo patching on distcc.

If I were doing this today, I'd probably focus on a specific setuid wrapper to escape the sandbox.
Comment 6 Kalin KOZHUHAROV 2017-01-07 18:26:22 UTC
OK, tested again.

Ran `watch -n1 'grep PROXY /var/tmp/portage/*/*/temp/environment'` while emerging a few ebuilds, no PROXY-related environment appeared. Also, no proxy-like processes were started. And no traffic on port 3632/tcp.

FYI:
# emerge --info |grep ^FEATURES
FEATURES="assume-digests binpkg-logs cgroup config-protect-if-modified distcc distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

# eselect python list
Available Python interpreters:
  [1]   python2.7
  [2]   python3.4 *

And switching to "-network-sandbox" works fine (for positive control).
Comment 7 Zac Medico gentoo-dev 2017-01-09 01:31:15 UTC
(In reply to Kalin KOZHUHAROV from comment #6)
> OK, tested again.
> 
> Ran `watch -n1 'grep PROXY /var/tmp/portage/*/*/temp/environment'` while
> emerging a few ebuilds, no PROXY-related environment appeared. Also, no
> proxy-like processes were started. And no traffic on port 3632/tcp.

It turns out that the PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables don't show up in the environment file because portage filters them out. I tested locally, and was able to observe distcc working with network-sandbox enabled.

I used a line like this in /etc/portage/bashrc to verify that the DISTCC_SOCKS_PROXY variable was set:

einfo DISTCC_SOCKS_PROXY=$DISTCC_SOCKS_PROXY

> And switching to "-network-sandbox" works fine (for positive control).

From the "Network is unreachable" messages that you got with DISTCC_VERBOSE=1, it seems like portage is not even setting up the proxy, and that should only happen if you're using python2.7. Please check `emerge --version` output confirm the python version being used. If it doesn't report Python 3.4.x, make sure that sys-apps/portage is built with PYTHON_TARGETS=python3_4 enabled.
Comment 8 Kalin KOZHUHAROV 2017-01-09 11:13:25 UTC
Created attachment 459342 [details]
emerge --info

OK, finally progress!

I tested again, caught the proxy process running and confirmed the envvar was set with the bashrc hack

However, still no packets flow to the server... Looking at the logs I found an error that was not clear before, I guess it is the core of my problem:

TypeError: must be str, not bytes
Task exception was never retrieved
future: <Task finished coro=<handle_proxy_conn() done, defined at /usr/lib/portage/python3.4/socks5-server.py:26> exception=TypeError('must be str, not bytes',)>
Traceback (most recent call last):
  File "/usr/lib64/python3.4/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "/usr/lib/portage/python3.4/socks5-server.py", line 100, in handle_proxy_conn
    addr, port)
  File "/usr/lib64/python3.4/asyncio/streams.py", line 78, in open_connection
    lambda: protocol, host, port, **kwds)
  File "/usr/lib64/python3.4/asyncio/base_events.py", line 630, in create_connection
    type=socket.SOCK_STREAM, proto=proto, flags=flags)
  File "/usr/lib64/python3.4/asyncio/base_events.py", line 574, in getaddrinfo
    info = _ipaddr_info(host, port, family, type, proto)
  File "/usr/lib64/python3.4/functools.py", line 472, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/lib64/python3.4/asyncio/base_events.py", line 113, in _ipaddr_info
    socket.inet_pton(af, host)

Attaching `emerge --info` FYR.
Comment 9 Zac Medico gentoo-dev 2017-01-09 20:23:04 UTC
Created attachment 459394 [details, diff]
bin/socks5-server.py: convert address from bytes to str

I can reproduce the "TypeError: must be str, not bytes" exception with python-3.4.5, and it does not occur with python-3.4.3-r1. Decoding the addr variable from bytes to str solves the problem.
Comment 10 Zac Medico gentoo-dev 2017-01-11 09:29:16 UTC
(In reply to Zac Medico from comment #9)
> Created attachment 459394 [details, diff] [details, diff]
> bin/socks5-server.py: convert address from bytes to str

Maybe the patch should assume 'idna' encoding rather than 'utf_8'. I think distcc just passes through whatever encoding exists in the config file.
Comment 11 Zac Medico gentoo-dev 2017-01-12 17:40:59 UTC
Created attachment 459804 [details, diff]
bin/socks5-server.py: convert address from bytes to str

Updated patch to use the idna codec. Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/c65af3933bff8c63c2ea9ab97be018d1
https://github.com/gentoo/portage/pull/87
Comment 13 Zac Medico gentoo-dev 2017-05-20 18:14:12 UTC
Fixed in portage-2.3.5.
Comment 14 Nikolay Kichukov 2019-01-12 23:15:10 UTC
Hi,
on portage 2.3.55 on 32bit arm which had the network-sandbox feature set and default python interpreter 2.7 this bug bit me so bad today. Hours were wasted trying to understand the change in behaviour and the 'failed to connect' and 'Network is unreachable' which just made no sense...

Thanks Kalin for addressing this issue with development and providing workarounds and the bug report.

I still depend on python2.7 to be the main interpreter on the system, so switching that globally is not an easy way out.
Comment 15 Zac Medico gentoo-dev 2019-01-15 21:19:26 UTC
(In reply to Nikolay Kichukov from comment #14)
> Hi,
> on portage 2.3.55 on 32bit arm which had the network-sandbox feature set and
> default python interpreter 2.7 this bug bit me so bad today. Hours were
> wasted trying to understand the change in behaviour and the 'failed to
> connect' and 'Network is unreachable' which just made no sense...
> 
> Thanks Kalin for addressing this issue with development and providing
> workarounds and the bug report.
> 
> I still depend on python2.7 to be the main interpreter on the system, so
> switching that globally is not an easy way out.

You can solve this by rebuilding sys-apps/portage with USE=python_targets_python2_7 disabled, like this:

echo sys-apps/portage -python_targets_python2_7 >> /etc/portage/package.use
emerge --oneshot --newuse sys-apps/portage

Also see bug 658172.
Comment 16 Larry the Git Cow gentoo-dev 2019-01-16 08:33:12 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=035582f0e31c071606635aac9cc4ba4b411612e7

commit 035582f0e31c071606635aac9cc4ba4b411612e7
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-01-14 08:11:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-01-16 07:48:59 +0000

    tests: add unit test for portage.util.socks5 (FEATURES=network-sandbox-proxy)
    
    Bug: https://bugs.gentoo.org/604474
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/util/test_socks5.py | 211 ++++++++++++++++++++++++++++++++++
 lib/portage/util/socks5.py            |  48 +++++++-
 2 files changed, 256 insertions(+), 3 deletions(-)