After bootstrapping from scratch the parallel-fetch feature is enabled. It seems that this feature somehow interacts with the use of an HTTP proxy, causing emerge operations from behind a proxy to fail. Workaround being tried for now: Append FEATURES="-parallel-fetch" to etc/make.conf. I am not sure how to pinpoint this problem. The problem appears when I try to do `emerge gkrellm'. This pulls in some 28 packages and for number 12 out of the 28, which is x11-libs/pixman, the fetching fails. Inspecting var/log/emerge-fetch.log it seems that wget tries to download that package without using the prescribed proxy. For other downloads logged in the same file the proxy is used. I'll be back with more info once I have verified that the workaround is a good one.
No, the workaround did not solve the problem. It does have an effect: Fetching is now sequential and no var/log/emerge-fetch.log file is produced. However, when trying to fetch x11-libs/pixman the configured HTTP proxy is no longer used. In the console log one can see close to 200 successful downloads, all using the proxy. The last successful one is dev-perl/XML-Parser. Then 9 seconds later an attempt to download x11-libs/pixman occurs, this time trying to connect directly to the Gentoo mirror. So, the mystery is, why does emerge suddenly stop using the proxy in the middle of the building of 28 required dependencies?
Please try the latest portage. zmedico potentially fixed this issue.
OK, great, I will undo my workaround. My next bootstrap-from-scratch runs in the early morning (CEST).
ok, is it a problem that the bootstrap image doesn't contain this newest portage?
Hmm, it sounds as if it will be a problem. What extra command or such should I insert so that I get the latest portage?
Well, I don't know how you do it now, as the bootstrap portage doesn't understand --jobs. If adding that option is a step after emerge --sync, then it should all be well.
This morning bootstrapping from scratch was run with no workarounds applied. The bootstrapping itself (following the webpage instructions) succeeds, but the resulting prefix tree has the problem described before: fetching distfiles using an HTTP proxy stops working at some point. At that point `emerge --version' reports: Portage 2.2.00.11225-prefix (default-prefix/linux/x86, gcc-4.2.4, unavailable, 2.6.16.53-0.16-smp i686) The packages that refuse to be fetched are pixman and glib. I will enclose emerge-fetch.log for inspection. Note that the failed fetch operations are followed by succesful fetch operations with later timestamps. `emerge --info' produces: Portage 2.2.00.11225-prefix (default-prefix/linux/x86, gcc-4.2.4, unavailable, 2.6.16.53-0.16-smp i686) ================================================================= System uname: Linux-2.6.16.53-0.16-smp-i686-Intel-R-_Core-TM-2_CPU_6600_@_2.40GHz-with-SuSE-10-i586 Timestamp of tree: Mon, 28 Jul 2008 00:53:18 +0000 ccache version 2.4 [disabled] app-shells/bash: 3.2_p39 dev-lang/python: 2.5.2-r5 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.61-r2 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18.50.0.7 sys-devel/gcc-config: 1.4.0-r04.5 sys-devel/libtool: 1.5.26 ACCEPT_KEYWORDS="~x86-linux" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-O2 -pipe" DISTDIR="/local/tmp/nightly/2008-07-28/usr/portage/distfiles" EPREFIX="/local/tmp/nightly/2008-07-28" FEATURES="collision-protect distlocks parallel-fetch preserve-libs sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://mirror.gentoo.no/ http://mirror.qubenet.net/mirror/gentoo/ http://mirror.muntinternet.net/pub/gentoo/ http://ftp.linux.ee/pub/gentoo/distfiles/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo" LANG="en_US" LDFLAGS="" MAKEOPTS="-j2" PKGDIR="/local/tmp/nightly/2008-07-28/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/local/scratch" PORTDIR="/local/tmp/nightly/2008-07-28/usr/portage" SYNC="rsync://rsync.prefix.freens.org/gentoo-portage-prefix" USE="X cracklib iconv midi mudflap ncurses nls openmp prefix readline ssl unicode x86 zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Created attachment 161518 [details] emerge-fetch.log The emerge-fetch.log file.
i have the same problem (there is a thread on gentoo-alt). as a temporary workaround, it works to 'mv wget{,.orig}' and create a script named wget to 'export http_proxy=...' to something usefull, and 'exec wget.orig "$@"'
I suspect this is a general portage problem, but I'm not sure. I'm most bothered by the "randomness" of the problem. It is reproducible, but it looks like suddenly the proxy is no longer used. Does anyone know from which portage version (revision) this no longer worked?
I can't help very much here because I have been caching distfiles locally. This reduces network traffic but of course it may disguise a problem like this one. I have also realized that the name of a distfile (including version and the -rNN tail) does not uniquely identify its content (an ebuild may be bugfixed without the -rNN number being raised). So, I have now turned off caching of distfiles. Everything is re-downloaded every time I do bootstrapping from scratch. And, it remains a mystery why "pixman" is treated specially in the 2008-07-28 tree. If I enter a bash shell in the tree and attempt `emerge pixman' manually I stumble over the same problem: the proxy is not used. If I instead try something trivial like `emerge man' (after ditching the man distfile first) the downloading works just as it should, using the proxy. Here is a workaround though: Append a line 'http_proxy=http://HOST:PORT/' to etc/wgetrc. Like the wget manpage says, this overrides the value of http_proxy in the environment. Having done this I can actually emerge pixman.
(In reply to comment #10) > I suspect this is a general portage problem, but I'm not sure. I'm most > bothered by the "randomness" of the problem. It is reproducible, but it looks > like suddenly the proxy is no longer used. > Does anyone know from which portage version (revision) this no longer worked? there is no randomness here. i see the problem all the time. for the revision: no plan, sorry... but i think portage-2.2.00.11187 still worked
It should be fixed in portage-2.2.00.11225 since the fix was in trunk r11223 and that got merged to the prefix branch in r11224.
But I am running portage-2.2.00.11225 already (see Comment #7), and the problem persists.
(In reply to comment #13) > It should be fixed in portage-2.2.00.11225 since the fix was in trunk r11223 > and that got merged to the prefix branch in r11224. hmm... seems to work again here with 11225... though i cannot remember when i did the last update here. i believe i had 11225 already when the problem came up.
(In reply to comment #15) > (In reply to comment #13) > ... > hmm... seems to work again here with 11225... though i cannot remember when i > did the last update here. i believe i had 11225 already when the problem came > up. Does it work for you to be behind a proxy and emerge the "pixman" package?
(2) mduft bin $ emerge -f --nodeps pixman >>> Building (1 of 1) x11-libs/pixman-0.11.8 for / >>> Downloading 'http://gentoo.inode.at/distfiles/pixman-0.11.8.tar.bz2' --2008-07-28 15:02:58-- http://gentoo.inode.at/distfiles/pixman-0.11.8.tar.bz2 Resolving www-proxy.salomon.at... 172.28.3.81 Connecting to www-proxy.salomon.at|172.28.3.81|:8080... connected. Proxy request sent, awaiting response... 200 OK Length: 360229 (352K) [application/octet-stream] Saving to: `/opt/gentoo.system/usr/portage/distfiles/pixman-0.11.8.tar.bz2' 100%[=================================================================================================================================================================>] 360,229 --.-K/s in 0.04s 2008-07-28 15:02:58 (8.58 MB/s) - `/opt/gentoo.system/usr/portage/distfiles/pixman-0.11.8.tar.bz2' saved [360229/360229] * pixman-0.11.8.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ] >>> Installing x11-libs/pixman-0.11.8 to / yes :)
Some further analysis of my partly-non-working tree of 2007-07-28: I moved usr/bin/wget to usr/bin/wget.bin and added a shellscript wrapper usr/bin/wget that execs wget.bin. The wrapper prints the environment variables to a file before doing the exec. And indeed, the environments are quite different depending on whether I choose to emerge "man" (representing emerge of a random Gentoo package) and "pixman". When emerging "man" I have 211 lines of environment, including lots of unrelated stuff that happened to be in the shell that I was using. This environment included the http_proxy definition. When emerging "pixman" I see a much smaller environment: 77 lines, all of them appear to be portage related. No http_proxy definition. It is as if a cleared environment has been set up, containing useful portage stuff but not including the http_proxy definition. Does this makes sense .. wget sometimes being given a cleared environment and sometimes a thicker one that inherits everything from the environment that invoked "emerge"? If that is OK then maybe all that is needed is to ensure that http_proxy is always inherited, even when the environment is otherwise cleared.
Created attachment 161575 [details, diff] make fetch() pass all config variables Here's one more spot that needed fixing.
I released zmedico's patch in portage-2.2.00.11248. @rabbe, it probably gets used in your nightly build. @mduft, if possible, manually test if the problem is really gone now.
My problem is solved now. `emerge gkrellm' builds all its dependencies successfully, which means that all wget operations through the proxy succeed.
since it worked here already again, i cannot verify. but i'll keep a 11225 around to see wether i can find a fetch problem, then upgrade and try again... so long, i guess this can be resolved.
@zmedico, do you want to have this bug on your tracker, if so please reassign, otherwise I'll close it.
I'll take it, thanks.
This is fixed in portage-2.2_rc5.
