Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 233103 - Parallel fetch and HTTP proxy do not go well together
Summary: Parallel fetch and HTTP proxy do not go well together
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: x86 Linux
: High normal (vote)
Assignee: Portage team
Depends on:
Blocks: 210077
  Show dependency tree
Reported: 2008-07-27 13:56 UTC by Rabbe Fogelholm
Modified: 2008-07-30 10:24 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

emerge-fetch.log (emerge-fetch.log,84.75 KB, text/plain)
2008-07-28 06:00 UTC, Rabbe Fogelholm
make fetch() pass all config variables (fetch_env.patch,513 bytes, patch)
2008-07-28 20:04 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Rabbe Fogelholm 2008-07-27 13:56:04 UTC
After bootstrapping from scratch the parallel-fetch feature is enabled. It seems that this feature somehow interacts with the use of an HTTP proxy, causing emerge operations from behind a proxy to fail.

Workaround being tried for now: Append FEATURES="-parallel-fetch" to etc/make.conf.

I am not sure how to pinpoint this problem. The problem appears when I try to do `emerge gkrellm'. This pulls in some 28 packages and for number 12 out of the 28, which is x11-libs/pixman, the fetching fails.

Inspecting var/log/emerge-fetch.log it seems that wget tries to download that package without using the prescribed proxy. For other downloads logged in the same file the proxy is used.

I'll be back with more info once I have verified that the workaround is a good one.
Comment 1 Rabbe Fogelholm 2008-07-27 16:35:10 UTC
No, the workaround did not solve the problem.

It does have an effect: Fetching is now sequential and no var/log/emerge-fetch.log file is produced. However, when trying to fetch x11-libs/pixman the configured HTTP proxy is no longer used.

In the console log one can see close to 200 successful downloads, all using the proxy. The last successful one is dev-perl/XML-Parser. Then 9 seconds later an attempt to download x11-libs/pixman occurs, this time trying to connect directly to the Gentoo mirror.

So, the mystery is, why does emerge suddenly stop using the proxy in the middle of the building of 28 required dependencies?
Comment 2 Fabian Groffen gentoo-dev 2008-07-27 18:42:38 UTC
Please try the latest portage.  zmedico potentially fixed this issue.
Comment 3 Rabbe Fogelholm 2008-07-27 18:58:47 UTC
OK, great, I will undo my workaround. My next bootstrap-from-scratch runs in the early morning (CEST).
Comment 4 Fabian Groffen gentoo-dev 2008-07-27 19:29:11 UTC
ok, is it a problem that the bootstrap image doesn't contain this newest portage?
Comment 5 Rabbe Fogelholm 2008-07-27 19:34:18 UTC
Hmm, it sounds as if it will be a problem. What extra command or such should I insert so that I get the latest portage?
Comment 6 Fabian Groffen gentoo-dev 2008-07-27 19:37:40 UTC
Well, I don't know how you do it now, as the bootstrap portage doesn't understand --jobs.  If adding that option is a step after emerge --sync, then it should all be well.
Comment 7 Rabbe Fogelholm 2008-07-28 05:57:25 UTC
This morning bootstrapping from scratch was run with no workarounds applied. The bootstrapping itself (following the webpage instructions) succeeds, but the resulting prefix tree has the problem described before: fetching distfiles using an HTTP proxy stops working at some point.

At that point `emerge --version' reports:

Portage (default-prefix/linux/x86, gcc-4.2.4, unavailable, i686)

The packages that refuse to be fetched are pixman and glib. I will enclose emerge-fetch.log for inspection. Note that the failed fetch operations are followed by succesful fetch operations with later timestamps.

`emerge --info' produces:

Portage (default-prefix/linux/x86, gcc-4.2.4, unavailable, i686)
System uname: Linux-
Timestamp of tree: Mon, 28 Jul 2008 00:53:18 +0000
ccache version 2.4 [disabled]
app-shells/bash:     3.2_p39
dev-lang/python:     2.5.2-r5
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.61-r2
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1
sys-devel/gcc-config: 1.4.0-r04.5
sys-devel/libtool:   1.5.26
CFLAGS="-O2 -pipe"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo"
CXXFLAGS="-O2 -pipe"
FEATURES="collision-protect distlocks parallel-fetch preserve-libs sandbox sfperms strict unmerge-orphans userfetch"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
USE="X cracklib iconv midi mudflap ncurses nls openmp prefix readline ssl unicode x86 zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU"

Comment 8 Rabbe Fogelholm 2008-07-28 06:00:04 UTC
Created attachment 161518 [details]

The emerge-fetch.log file.
Comment 9 Markus Duft (RETIRED) gentoo-dev 2008-07-28 06:25:26 UTC
i have the same problem (there is a thread on gentoo-alt). as a temporary workaround, it works to 'mv wget{,.orig}' and create a script named wget to 'export http_proxy=...' to something usefull, and 'exec wget.orig "$@"'
Comment 10 Fabian Groffen gentoo-dev 2008-07-28 10:38:01 UTC
I suspect this is a general portage problem, but I'm not sure.  I'm most bothered by the "randomness" of the problem.  It is reproducible, but it looks like suddenly the proxy is no longer used.

Does anyone know from which portage version (revision) this no longer worked?
Comment 11 Rabbe Fogelholm 2008-07-28 11:41:15 UTC
I can't help very much here because I have been caching distfiles locally. This reduces network traffic but of course it may disguise a problem like this one.

I have also realized that the name of a distfile (including version and the -rNN tail) does not uniquely identify its content (an ebuild may be bugfixed without the -rNN number being raised).

So, I have now turned off caching of distfiles. Everything is re-downloaded every time I do bootstrapping from scratch.

And, it remains a mystery why "pixman" is treated specially in the 2008-07-28 tree. If I enter a bash shell in the tree and attempt `emerge pixman' manually I stumble over the same problem: the proxy is not used. If I instead try something trivial like `emerge man' (after ditching the man distfile first) the downloading works just as it should, using the proxy.

Here is a workaround though: Append a line 'http_proxy=http://HOST:PORT/' to etc/wgetrc. Like the wget manpage says, this overrides the value of http_proxy in the environment. Having done this I can actually emerge pixman.
Comment 12 Markus Duft (RETIRED) gentoo-dev 2008-07-28 11:54:44 UTC
(In reply to comment #10)
> I suspect this is a general portage problem, but I'm not sure.  I'm most
> bothered by the "randomness" of the problem.  It is reproducible, but it looks
> like suddenly the proxy is no longer used.
> Does anyone know from which portage version (revision) this no longer worked?

there is no randomness here. i see the problem all the time. for the revision: no plan, sorry... but i think portage- still worked
Comment 13 Zac Medico gentoo-dev 2008-07-28 12:13:52 UTC
It should be fixed in portage- since the fix was in trunk r11223 and that got merged to the prefix branch in r11224.
Comment 14 Rabbe Fogelholm 2008-07-28 12:20:58 UTC
But I am running portage- already (see Comment #7), and the problem persists.
Comment 15 Markus Duft (RETIRED) gentoo-dev 2008-07-28 12:23:41 UTC
(In reply to comment #13)
> It should be fixed in portage- since the fix was in trunk r11223
> and that got merged to the prefix branch in r11224.

hmm... seems to work again here with 11225... though i cannot remember when i did the last update here. i believe i had 11225 already when the problem came up.
Comment 16 Rabbe Fogelholm 2008-07-28 12:59:32 UTC
(In reply to comment #15)
> (In reply to comment #13)
> ...
> hmm... seems to work again here with 11225... though i cannot remember when i
> did the last update here. i believe i had 11225 already when the problem came
> up.

Does it work for you to be behind a proxy and emerge the "pixman" package?

Comment 17 Markus Duft (RETIRED) gentoo-dev 2008-07-28 13:03:41 UTC
(2) mduft bin $ emerge -f --nodeps pixman

>>> Building (1 of 1) x11-libs/pixman-0.11.8 for /
>>> Downloading ''
--2008-07-28 15:02:58--
Connecting to||:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 360229 (352K) [application/octet-stream]
Saving to: `/opt/gentoo.system/usr/portage/distfiles/pixman-0.11.8.tar.bz2'

100%[=================================================================================================================================================================>] 360,229     --.-K/s   in 0.04s   

2008-07-28 15:02:58 (8.58 MB/s) - `/opt/gentoo.system/usr/portage/distfiles/pixman-0.11.8.tar.bz2' saved [360229/360229]

 * pixman-0.11.8.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...                                                                                 [ ok ]
>>> Installing x11-libs/pixman-0.11.8 to /

yes :)
Comment 18 Rabbe Fogelholm 2008-07-28 13:53:11 UTC
Some further analysis of my partly-non-working tree of 2007-07-28: I moved usr/bin/wget to usr/bin/wget.bin and added a shellscript wrapper usr/bin/wget that execs wget.bin.

The wrapper prints the environment variables to a file before doing the exec. And indeed, the environments are quite different depending on whether I choose to emerge "man" (representing emerge of a random Gentoo package) and "pixman".

When emerging "man" I have 211 lines of environment, including lots of unrelated stuff that happened to be in the shell that I was using. This environment included the http_proxy definition.

When emerging "pixman" I see a much smaller environment: 77 lines, all of them appear to be portage related. No http_proxy definition. It is as if a cleared environment has been set up, containing useful portage stuff but not including the http_proxy definition.

Does this makes sense .. wget sometimes being given a cleared environment and sometimes a thicker one that inherits everything from the environment that invoked "emerge"?

If that is OK then maybe all that is needed is to ensure that http_proxy is always inherited, even when the environment is otherwise cleared.
Comment 19 Zac Medico gentoo-dev 2008-07-28 20:04:45 UTC
Created attachment 161575 [details, diff]
make fetch() pass all config variables

Here's one more spot that needed fixing.
Comment 20 Fabian Groffen gentoo-dev 2008-07-28 20:31:18 UTC
I released zmedico's patch in portage-
@rabbe, it probably gets used in your nightly build.
@mduft, if possible, manually test if the problem is really gone now.
Comment 21 Rabbe Fogelholm 2008-07-29 05:59:25 UTC
My problem is solved now. `emerge gkrellm' builds all its dependencies successfully, which means that all wget operations through the proxy succeed.
Comment 22 Markus Duft (RETIRED) gentoo-dev 2008-07-29 06:14:20 UTC
since it worked here already again, i cannot verify. but i'll keep a 11225 around to see wether i can find a fetch problem, then upgrade and try again... so long, i guess this can be resolved.
Comment 23 Fabian Groffen gentoo-dev 2008-07-29 09:43:33 UTC
@zmedico, do you want to have this bug on your tracker, if so please reassign, otherwise I'll close it.
Comment 24 Zac Medico gentoo-dev 2008-07-29 09:49:27 UTC
I'll take it, thanks.
Comment 25 Zac Medico gentoo-dev 2008-07-30 10:24:27 UTC
This is fixed in portage-2.2_rc5.