emerge --sync with sync-type=webrsync is ignoring proxy settings in /etc/portage/make.conf I have many systems that used emerge-webrsync successfully for years, reading their proxy settings from make.conf (some but not all systems also have a proxy usable for regular users, listed in /etc/env.d/99proxy and/or profile.d/proxy.sh), with FEATURES='webrsync-gpg' enabled, and an /etc/portage/gpg/ directory that I managed (imperfectly?) myself. Due to the recent changes that deprecated emerge-webrsync, I've been updating repos.conf/gentoo.conf to add 'sync-webrsync-verify-signature = true', removing FEATURES='webrsync-gpg' from make.conf, and switching cron jobs from emerge-webrsync to emerge --sync. And all of those cron jobs are failing, because emerge --sync is ignoring the proxy settings in make.conf, and is not processing environment files. portageq envvar -v http_proxy and https_proxy show the expected results (reading from make.conf), yet emerge --sync does not use them. It looks like gemato passes proxy settings to gnupg just fine iff it is given proxy settings to pass along. So emerge --sync in a shell on such a system works fine as long as env.d/ or profile.d/ files set and export the variables, of course. But that doesn't help for cron jobs, or on systems that don't have proxies configured for regular user shells. An ugly hack that works for now: change emerge --sync cron job to: . /etc/portage/make.conf && export http_proxy && /usr/bin/emerge --sync ...but that does not seem like it should be necessary.
(In reply to Hank Leininger from comment #0) > portageq envvar -v http_proxy and https_proxy show the expected results > (reading from make.conf), yet emerge --sync does not use them. This suggests that there's an overriding http_proxy setting in the cron environment. If you create a cron job that directs the output of /usr/bin/env to a file, do you see an overriding http_proxy in there?
> This suggests that there's an overriding http_proxy setting in the cron > environment. If you create a cron job that directs the output of /usr/bin/env > to a file, do you see an overriding http_proxy in there? Nope. They are not set at all for cron jobs: # set | egrep -i proxy | sed -E 's/[0-9.]+:[0-9]+/1.2.3.4:5678/' RSYNC_PROXY=1.2.3.4:5678 ftp_proxy=http://1.2.3.4:5678/ http_proxy=http://1.2.3.4:5678/ https_proxy=http://1.2.3.4:5678/ rsync_proxy=1.2.3.4:5678 # egrep -i proxy /etc/portage/make.conf | sed -E 's/[0-9.]+:[0-9]+/1.2.3.4:5678/' http_proxy=http://1.2.3.4:5678/ https_proxy=http://1.2.3.4:5678/ ftp_proxy=http://1.2.3.4:5678/ rsync_proxy=1.2.3.4:5678 RSYNC_PROXY=1.2.3.4:5678 # crontab -l | tail -n1 * * * * * /usr/bin/env >/root/rootenv # egrep -i proxy /root/rootenv # It's typical for cron jobs to launch w/o fully populated environments, IIRC because they don't inherit them from crond and jobs are not launched as login shells. But emerge-webrsync used to read from /etc/portage/make.conf, and emerge --sync does not.
I've locally modified my emerge-webrsync to echo the http_proxy setting, and I can see the http_proxy setting from my /etc/portage/make.conf (both before and after the portageq call). So, I do not know how to explain why a http_proxy setting from your /etc/portage/make.conf would be ineffective. For emerge --sync with sync-type=webrsync, the code for bug 185048 is now redundant because emerge exports all of the make.conf variables so that they're available even before the portageq call. This means that https_proxy should also work even though it's not included here: https://gitweb.gentoo.org/proj/portage.git/commit/?id=e8517a23f57fed2291aa6a4a65b791d7b819510a
> I've locally modified my emerge-webrsync to echo the http_proxy setting, and I > can see the http_proxy setting from my /etc/portage/make.conf (both before and > after the portageq call). So, I do not know how to explain why a http_proxy > setting from your /etc/portage/make.conf would be ineffective. OK, that's consistent - like I said, emerge-webrsync has always worked, to pick up settings from make.conf. > For emerge --sync with sync-type=webrsync, the code for bug 185048 is now > redundant because emerge exports all of the make.conf variables so that they're > available even before the portageq call. This means that https_proxy should > also work even though it's not included here: In my case it's make.conf's http_proxy that matters (using an http URL so that the proxy can cache it). And it's not being picked up by emerge --sync. Simulating that: # unset RSYNC_PROXY ftp_proxy http_proxy https_proxy rsync_proxy # set | egrep -i proxy= # # egrep -i proxy /etc/portage/make.conf | sed -E 's/[0-9.]+:[0-9]+/1.2.3.4:5678/' http_proxy=http://1.2.3.4:5678/ https_proxy=http://1.2.3.4:5678/ ftp_proxy=http://1.2.3.4:5678/ rsync_proxy=1.2.3.4:5678 RSYNC_PROXY=1.2.3.4:5678 # set | egrep -i proxy= # # emerge --sync >>> Syncing repository 'gentoo' into '/usr/portage'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD ... [ !! ] * Refreshing keys from keyserver hkps://keys.gentoo.org ...OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: No keyserver available OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: No keyserver available ... It just fails and fails, and eventually gives up. I ran with bash -x and strace and such earlier, and confirmed emerge --sync and the gemato module will use an http_proxy that it finds already set in the environment, but will not read it out of /etc/portage/make.conf I mentioned that portageq produces the expected results to show that make.conf is parsed correctly by it. I don't have any indication that emerge --sync w/sync-type=webrsync is using portageq (and/or parsing those variables out of make.conf before calling gemato), in fact the opposite, it looks like it isn't, or this would be working. And by the way thank you for looking into this :)
The issue is that the proxy variables have to be environment variables of the emerge process in order for gemato to pick them up (from os.environ). The gemato OpenPGPEnvironment does not provide any other way to pass in these environment variables.
There's no formal standard, but it looks both uppercase and lowercase versions of these variables are commonly supported: all_proxy ftp_proxy http_proxy https_proxy no_proxy
(In reply to Zac Medico from comment #5) > The issue is that the proxy variables have to be environment variables of > the emerge process in order for gemato to pick them up (from os.environ). > The gemato OpenPGPEnvironment does not provide any other way to pass in > these environment variables. It feels like a reasonable thing to do would be to parse make.conf settings (maybe just the proxy ones specifically) and stick them into the environment before handing off to gemato? emerge $packagename respects proxy settings in make.conf, emerge --sync / gemato ought to also. I don't know how that'd fit with the code as it currently is, but IMHO it seems like it'd be the most "do what I mean" and consistent with other documented behavior.
I think this one is fixed given the commits in bug 740904 and https://bugs.gentoo.org/745771#c0. I've got some more bits I'm going to push in a minute though for https_proxy.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2410f8d7dde2ac12de4ec040696bcbb51e09ce9f commit 2410f8d7dde2ac12de4ec040696bcbb51e09ce9f Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-19 13:54:08 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-19 14:49:34 +0000 save-ebuild-env.sh, special_env_vars.py: add https_proxy For parity with http_proxy. Bug: https://bugs.gentoo.org/691434 Bug: https://bugs.gentoo.org/835927 Bug: https://bugs.gentoo.org/911629 Signed-off-by: Sam James <sam@gentoo.org> bin/save-ebuild-env.sh | 2 +- lib/portage/package/ebuild/_config/special_env_vars.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=eb3a3878e36e32ed0f8385b7ff8e978fbbe1aff3 commit eb3a3878e36e32ed0f8385b7ff8e978fbbe1aff3 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-19 13:51:45 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-19 14:49:34 +0000 sync: handle https_proxy too Pass down https_proxy into gemato. If http_proxy isn't set, but https_proxy is, then let's fall back to it to help users out. gnupg itself doesn't use https_proxy but users may have only https_proxy set rather than http_proxy and https_proxy. We've had cases reported where users only have https_proxy and not http_proxy set, so include this to help them out too. Bug: https://bugs.gentoo.org/691434 Bug: https://bugs.gentoo.org/835927 Bug: https://bugs.gentoo.org/911629 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/sync/syncbase.py | 2 ++ 1 file changed, 2 insertions(+) https://gitweb.gentoo.org/proj/portage.git/commit/?id=aa37e3993866f1d75e855f8c7300749fadd846af commit aa37e3993866f1d75e855f8c7300749fadd846af Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-19 13:49:53 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-19 14:49:33 +0000 emerge-webrsync: pass --proxy to gemato if http_proxy or https_proxy is set Make sure we pass proxy settings down into gemato. We include a fallback w/ https_proxy, but note that gnupg doesn't actually use https_proxy but we've had cases reported where users only have https_proxy and not http_proxy set, so include this to help them out too. Bug: https://bugs.gentoo.org/691434 Bug: https://bugs.gentoo.org/835927 Bug: https://bugs.gentoo.org/911629 Signed-off-by: Sam James <sam@gentoo.org> bin/emerge-webrsync | 7 +++++++ 1 file changed, 7 insertions(+) https://gitweb.gentoo.org/proj/portage.git/commit/?id=f0b44fc1cc67114662b84905bc1e0cd9ce1bba28 commit f0b44fc1cc67114662b84905bc1e0cd9ce1bba28 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-19 13:49:19 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-19 14:49:33 +0000 emerge-webrsync: handle https_proxy too If `https_proxy` is also set in make.conf, pick that up so it's passed down to wget. Bug: https://bugs.gentoo.org/691434 Bug: https://bugs.gentoo.org/835927 Bug: https://bugs.gentoo.org/911629 Signed-off-by: Sam James <sam@gentoo.org> bin/emerge-webrsync | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a6a68c0255b6d3132dd555b1519a035204bffde2 commit a6a68c0255b6d3132dd555b1519a035204bffde2 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-20 07:57:47 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-20 07:58:22 +0000 sys-apps/portage: add 3.0.51 Bug: https://bugs.gentoo.org/587484 Closes: https://bugs.gentoo.org/673624 Closes: https://bugs.gentoo.org/691434 Closes: https://bugs.gentoo.org/835927 Closes: https://bugs.gentoo.org/909009 Closes: https://bugs.gentoo.org/911335 Closes: https://bugs.gentoo.org/911629 Closes: https://bugs.gentoo.org/911938 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.51.ebuild | 234 +++++++++++++++++++++++++++++++++ 2 files changed, 235 insertions(+)
