Hi, I'm currently setting up a portage binhost server for the company I'm working for and I found the following problem when our clients try to access the server via http or https: testsys ~ # egrep "^(EMERGE_DEFAULT_OPTS|PORTAGE_BINHOST)" /etc/make.conf PORTAGE_BINHOST='https://user:password@192.168.10.107/amd64/desktop/kde' EMERGE_DEFAULT_OPTS="--alphabetical --with-bdeps=y --misspell-suggestions=n --autounmask=n --quiet-build=n -G -K --binpkg-respect-use=y" testsys ~ # eselect python list Available Python interpreters: [1] python2.7 [2] python3.2 * testsys ~ # emerge -uDav @world Traceback (most recent call last): File "/usr/lib64/python3.2/http/client.py", line 679, in _set_hostport port = int(host[i+1:]) ValueError: invalid literal for int() with base 10: 'password@192.168.10.107' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/emerge", line 43, in <module> retval = emerge_main() File "/usr/lib64/portage/pym/_emerge/main.py", line 1713, in emerge_main getbinpkgs="--getbinpkg" in myopts) File "/usr/lib64/portage/pym/portage/dbapi/bintree.py", line 582, in populate self._populate(getbinpkgs) File "/usr/lib64/portage/pym/portage/dbapi/bintree.py", line 846, in _populate f = urllib_request_urlopen(url) File "/usr/lib64/python3.2/urllib/request.py", line 138, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.2/urllib/request.py", line 369, in open response = self._open(req, data) File "/usr/lib64/python3.2/urllib/request.py", line 387, in _open '_open', req) File "/usr/lib64/python3.2/urllib/request.py", line 347, in _call_chain result = func(*args) File "/usr/lib64/python3.2/urllib/request.py", line 1172, in https_open context=self._context, check_hostname=self._check_hostname) File "/usr/lib64/python3.2/urllib/request.py", line 1107, in do_open h = http_class(host, timeout=req.timeout, **http_conn_args) File "/usr/lib64/python3.2/http/client.py", line 1075, in __init__ source_address) File "/usr/lib64/python3.2/http/client.py", line 658, in __init__ self._set_hostport(host, port) File "/usr/lib64/python3.2/http/client.py", line 681, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) http.client.InvalidURL: nonnumeric port: 'password@192.168.10.107' testsys ~ # As you can see portage is stumbling about the "user:password" part of the PORTAGE_BINHOST URI. This happens with both http and https but not with ftp. As soon as I remove "user:password" from the URI http and https do works as expected. Unfortunately we need restricted https access for some of our external servers.
I forgot to mention, this only happens when python-3.2 is set via "eselect python". The problem is gone once python-2.7 is set via eselect.
Currently Portage uses urllib.urlopen() with Python 2 and urllib.request.urlopen() with Python 3. urllib.request module of Python 3 corresponds to urllib2 module of Python 2. urllib2.urlopen() has the same problem: $ python2.7 -c 'import urllib2; urllib2.urlopen("http://user:password@gentoo.org")' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 400, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 418, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 378, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1207, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib64/python2.7/urllib2.py", line 1146, in do_open h = http_class(host, timeout=req.timeout) # will parse host:port File "/usr/lib64/python2.7/httplib.py", line 693, in __init__ self._set_hostport(host, port) File "/usr/lib64/python2.7/httplib.py", line 721, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: 'password@gentoo.org'
Created attachment 310549 [details, diff] portage-urlopen.patch This patch removes usage of deprecated urllib.urlopen() and probably fixes usage of urlopen() from new modules. Please test this patch with Python 2 and 3. I can commit it if it works.
The previous patch wasn't working with URLs, which require authentication, probably due to realm=None. The new patch explicitly uses HTTPPasswordMgrWithDefaultRealm instead of HTTPPasswordMgr. I have successfully tested portage.util.urlopen() on localhost:8080/manage (from Zope), which requires authentication. Commit: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commitdiff;h=0a9cc38a66ded0cf0e5b534cb24b970fc9c21920 You might test sys-apps/portage-9999.
This is released for testing in 2.2.0_alpha102.
This is fixed in 2.1.10.58.
Since the code for this bug doesn't work with all protocols, I've made it so that it's only used as a fallback: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=2c191d17c5c879c51aeb1c6043d9185b9a5887c0
