Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 824782 - dev-vcs/git-2.34.0: "Cannot fetch origin" (error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504) for anongit.gentoo.org/git/repo/sync/gentoo.git
Summary: dev-vcs/git-2.34.0: "Cannot fetch origin" (error: RPC failed; HTTP 504 curl 2...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Git (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-11-19 09:23 UTC by peter@prh.myzen.co.uk
Modified: 2023-12-24 17:53 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info (emerge.info,6.02 KB, application/x-info)
2021-11-19 09:23 UTC, peter@prh.myzen.co.uk
Details

Note You need to log in before you can comment on or make changes to this bug.
Description peter@prh.myzen.co.uk 2021-11-19 09:23:23 UTC
Created attachment 752838 [details]
emerge --info

On running 'emerge --sync' I get this error:

/usr/bin/git fetch origin
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: the remote end hung up unexpectedly
!!! git fetch error in /var/db/repos/gentoo

If I delete /var/db/repos/gentoo and try again, it runs to completion, but then the next sync fails in the same way. I've masked this version of git for the time being.

This is my /etc/portage/repos.conf/gentoo.conf:

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /var/db/repos/gentoo
auto-sync = yes

sync-type = git
clone-depth = 1
sync-uri = https://anongit.gentoo.org/git/repo/sync/gentoo.git

I tried chown -R portage:portage /var/db/repos/gentoo but it didn't help.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-11-19 09:58:52 UTC
Well, as a workaround I'd suggest syncing from github (https://github.com/gentoo-mirror/gentoo).
Comment 2 peter@prh.myzen.co.uk 2021-11-19 12:06:25 UTC
(In reply to Michał Górny from comment #1)
> Well, as a workaround I'd suggest syncing from github
> (https://github.com/gentoo-mirror/gentoo).

I could try that, but for the moment I've reverted to the last stable version: 2.32.0-r1.
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-19 12:09:06 UTC
(In reply to peter@prh.myzen.co.uk from comment #2)
> (In reply to Michał Górny from comment #1)
> > Well, as a workaround I'd suggest syncing from github
> > (https://github.com/gentoo-mirror/gentoo).
> 
> I could try that, but for the moment I've reverted to the last stable
> version: 2.32.0-r1.

The git version really makes a difference?
Comment 4 peter@prh.myzen.co.uk 2021-11-19 12:21:03 UTC
(In reply to Sam James from comment #3)

> The git version really makes a difference?

Yes, it's repeatable. This is on a system newly built from scratch, using the desktop stage 3 and ~amd64 throughout.
Comment 5 jospezial 2022-01-02 04:10:39 UTC
Confirming.

 * Fetching https://gitlab.freedesktop.org/xorg/xserver.git ...
git fetch https://gitlab.freedesktop.org/xorg/xserver.git +HEAD:refs/git-r3/HEAD
fatal: unable to access 'https://gitlab.freedesktop.org/xorg/xserver.git/': The requested URL returned error: 504
 * ERROR: x11-base/xorg-server-9999::gentoo failed (unpack phase):
 *   Unable to fetch from any of EGIT_REPO_URI

dev-vcs/git-2.34.1-r1::gentoo was built with the following:
USE="blksha1 curl gpg highlight iconv nls pcre perl threads webdav xinetd -cgi -cvs -doc -emacs -gnome-keyring -mediawiki -mediawiki-experimental -perforce (-ppcsha1) -subversion -test -tk" ABI_X86="(64)" PYTHON_SINGLE_TARGET="python3_9 -python3_10 -python3_8"
FEATURES="qa-unresolved-soname-deps usersandbox ebuild-locks merge-sync binpkg-docompress strict parallel-fetch multilib-strict unmerge-orphans unknown-features-warn binpkg-logs ccache usersync buildpkg-live network-sandbox news unmerge-logs userpriv userfetch xattr ipc-sandbox pid-sandbox protect-owned distlocks binpkg-dostrip assume-digests sfperms config-protect-if-modified preserve-libs sandbox binpkg-multi-instance fixlafiles


Failed sporadic on a few packages of my live-rebuild.
I wonder why I have not seen these errors before today.
6. Dez 17:37 /usr/bin/git
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-01-02 04:27:27 UTC
(In reply to jospezial from comment #5)
> Confirming.
> 
>  * Fetching https://gitlab.freedesktop.org/xorg/xserver.git ...
> git fetch https://gitlab.freedesktop.org/xorg/xserver.git
> +HEAD:refs/git-r3/HEAD
> fatal: unable to access 'https://gitlab.freedesktop.org/xorg/xserver.git/':
> The requested URL returned error: 504

Note that gitlab.fd.o has been having persistent issues recently.
Comment 7 peter@prh.myzen.co.uk 2022-01-02 09:16:09 UTC
I was syncing with anongit.gentoo.org/git/repo/sync/gentoo.git. (I don't know where that idea came from.) The problem went away after I reverted to github.com/gentoo-mirror/gentoo, as Michael suggested in Comment 1.
Comment 8 peter@prh.myzen.co.uk 2022-01-02 09:54:46 UTC
One other detail: after changing the sync host, I had to delete my local tree once more before the git sync worked cleanly.

Therefore I deduce that the other sync host was leaving something harmful in my local tree. I can't say what that might have been.
Comment 9 jospezial 2022-01-02 16:27:18 UTC
(In reply to Sam James from comment #6)
> (In reply to jospezial from comment #5)
> > Confirming.
> > 
> >  * Fetching https://gitlab.freedesktop.org/xorg/xserver.git ...
> > git fetch https://gitlab.freedesktop.org/xorg/xserver.git
> > +HEAD:refs/git-r3/HEAD
> > fatal: unable to access 'https://gitlab.freedesktop.org/xorg/xserver.git/':
> > The requested URL returned error: 504
> 
> Note that gitlab.fd.o has been having persistent issues recently.

failing live packages:
x11-libs/libvdpau x11-base/xorg-server x11-drivers/xf86-video-amdgpu x11-drivers/xf86-video-ati
All on https://gitlab.freedesktop.org...
Fetch hangs until timeout.

I also can't "git clone" these repos but I can open the URLs in firefox.
Downgrade to dev-vcs/git-2.32.0-r1 does not help.

So it is the gitlab.freedesktop.org server to blame.
Comment 10 Marcin Mirosław 2022-01-02 17:00:36 UTC
This is due to OOM killing git while fetching phase. I have also such problem with other repos. Especially with "-depth=1".
 git is eating above 6GB on my box before OOM is killing it.
Comment 11 jospezial 2022-01-02 20:00:11 UTC
(In reply to Marcin Mirosław from comment #10)
> This is due to OOM killing git while fetching phase. I have also such
> problem with other repos. Especially with "-depth=1".
>  git is eating above 6GB on my box before OOM is killing it.

No memory eating here. So no oom in syslog for that time.
Comment 12 cyrillic 2022-01-04 04:14:46 UTC
(In reply to Sam James from comment #6)
> Note that gitlab.fd.o has been having persistent issues recently.

I am having the same problem, but it is not all of gitlab.freedesktop.org it is only the xorg section. Other sections like mesa and gstreamer are working fine.
Comment 13 jospezial 2022-01-04 12:09:06 UTC
Fetching from gitlab.freedesktop.org works normal again since yesterday.
Comment 14 peter@prh.myzen.co.uk 2022-01-08 12:59:59 UTC
After building a new stable system from scratch, I now get the problem again with sync-uri = https://github.com/gentoo-mirror/gentoo
Comment 15 jospezial 2022-01-09 06:04:23 UTC
(In reply to jospezial from comment #13)
> Fetching from gitlab.freedesktop.org works normal again since yesterday.

gitlab.freedesktop.org is unreliable.
Ability to fetch or clone changes from second to second and from try to try.

I had no problems syncing the portage tree.
sync-uri = https://anongit.gentoo.org/git/repo/gentoo.git
Comment 16 David Sardari 2022-03-31 16:49:27 UTC
Setting "sync-depth = 1" solved it for me.
Comment 17 David Sardari 2022-03-31 20:35:45 UTC
Initially (empty /var/db/repos/gentoo), portage causes the execution of:
➤ git clone --depth 1 https://github.com/gentoo-mirror/gentoo.git .



In /var/db/repos/gentoo, the situation is then as follows:
➤ git log | tee
commit 7e1a50ba72299ad8d9ec7ac71030cf84eb9b9e1a
Author: Repository mirror & CI <repomirrorci@gentoo.org>
Date:   Thu Mar 31 19:19:46 2022 +0000

    2022-03-31 19:19:43 UTC
➤ git rev-list --count HEAD
1



Later on, additional commits are pushed:
➤ curl -fsSL --proto '=https' --tlsv1.3 https://api.github.com/repos/gentoo-mirror/gentoo | jq '.pushed_at'
"2022-03-31T19:49:52Z"



A repeated "emerge --fetch" basically caused the execution of:
➤ time git fetch origin
remote: Enumerating objects: 3509233, done.
remote: Counting objects: 100% (3509187/3509187), done.
remote: Compressing objects: 100% (1128385/1128385), done.
remote: Total 3476530 (delta 2406289), reused 3411413 (delta 2345631), pack-reused 0
Receiving objects: 100% (3476530/3476530), 1.11 GiB | 6.65 MiB/s, done.
Resolving deltas: 100% (2406289/2406289), completed with 15695 local objects.
From https://github.com/gentoo-mirror/gentoo
   7e1a50ba722..e769d5bd95d  stable     -> origin/stable

________________________________________________________
Executed in  447.51 secs    fish           external
   usr time  109.90 secs  548.00 micros  109.90 secs
   sys time   11.77 secs  388.00 micros   11.77 secs
➤ git rev-list --count HEAD
561191



As you can see, this is very time-consuming and most likely very ressource-consuming for GitHub to take so long. The reason lies in the fetch of the whole git history (561190 commits in total). This requires the enumeration, counting and compression of lots of objects.

CONCLUSION: For this procedure to work with https://anongit.gentoo.org/git/repo/sync/gentoo.git, sufficient ressources have to be provided to the server and timeout/quota settings need to be checked. But, a better approach would be fetching with "--depth 1". This can be done by setting "sync-depth = 1" to /usr/share/portage/config/repos.conf. IMHO, a better approach would be making portage behave the same way for the initial clone and the following fetches. I created a PR for this purpose: https://github.com/gentoo/portage/pull/800
Comment 18 David Sardari 2022-04-04 04:07:42 UTC
As I am not getting anywhere with https://github.com/gentoo/portage/pull/801, I decided to write a non squashfs/overlayfs solution:
https://github.com/duxsco/gentoo-git

I leave the PR as is and take care of other stuff 🙂
Comment 19 Larry the Git Cow gentoo-dev 2022-10-18 00:41:39 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f2207e41792d8180519e5695934af05daad3c971

commit f2207e41792d8180519e5695934af05daad3c971
Author:     David Sardari <d@duxsco.de>
AuthorDate: 2022-03-31 20:29:54 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-10-18 00:41:35 +0000

    GitSync.update: default to "--depth 1" (bug 824782 comment 17)
    
    Enforce the use of "--depth" in both GitSync.new and GitSync.update following the same logic.
    
    Each function gets its own designated portage option of "clone-depth" and "sync-depth".
    This means that portage option "sync-depth" is not taken into consideration anymore in GitSync.new.
    
    Portage option "clone-depth" and "sync-depth" lead to following behaviour with a value of:
    - 0: The use of "--depth <value>" is disabled causing the retrieval of the whole Git history.
    - ℕ (positive integer): "--depth <value>" is used.
    - Unset portage option: "--depth 1" is used.
    
    Bug: https://bugs.gentoo.org/824782
    Signed-off-by: David Sardari <d@duxsco.de>
    Closes: https://github.com/gentoo/portage/pull/801
    Signed-off-by: Sam James <sam@gentoo.org>

 NEWS                                |  3 +++
 lib/portage/sync/modules/git/git.py | 14 ++++++++------
 man/portage.5                       |  8 +++++---
 3 files changed, 16 insertions(+), 9 deletions(-)
Comment 20 jospezial 2022-10-19 08:33:55 UTC
(In reply to Larry the Git Cow from comment #19)
> The bug has been referenced in the following commit(s):
> 
> https://gitweb.gentoo.org/proj/portage.git/commit/
> ?id=f2207e41792d8180519e5695934af05daad3c971
> 
> commit f2207e41792d8180519e5695934af05daad3c971
> Author:     David Sardari <d@duxsco.de>
> AuthorDate: 2022-03-31 20:29:54 +0000
> Commit:     Sam James <sam@gentoo.org>
> CommitDate: 2022-10-18 00:41:35 +0000
> 
>     GitSync.update: default to "--depth 1" (bug 824782 comment 17)
>     
>     Enforce the use of "--depth" in both GitSync.new and GitSync.update
> following the same logic.
>     
>     Each function gets its own designated portage option of "clone-depth"
> and "sync-depth".
>     This means that portage option "sync-depth" is not taken into
> consideration anymore in GitSync.new.
>     
>     Portage option "clone-depth" and "sync-depth" lead to following
> behaviour with a value of:
>     - 0: The use of "--depth <value>" is disabled causing the retrieval of
> the whole Git history.
>     - ℕ (positive integer): "--depth <value>" is used.
>     - Unset portage option: "--depth 1" is used.
>     
>     Bug: https://bugs.gentoo.org/824782
>     Signed-off-by: David Sardari <d@duxsco.de>
>     Closes: https://github.com/gentoo/portage/pull/801
>     Signed-off-by: Sam James <sam@gentoo.org>
> 
>  NEWS                                |  3 +++
>  lib/portage/sync/modules/git/git.py | 14 ++++++++------
>  man/portage.5                       |  8 +++++---
>  3 files changed, 16 insertions(+), 9 deletions(-)

After that commit emerge -vvv --sync does not show me the changed files of the git syncs.
Comment 21 David Sardari 2022-10-27 15:38:31 UTC
Can you try with "clone-depth" and "sync-depth" being set to 2 or 0? I personally use eix-sync to get changes shown.
Comment 22 Matthew Marchese Gentoo Infrastructure gentoo-dev 2022-11-05 00:30:21 UTC
This may or may not be useful for this issue, but I'm including it in case it's helpful.

$ time GIT_TRACE=1 git fetch --verbose origin
16:37:04.288191 git.c:460               trace: built-in: git fetch --verbose origin
16:37:04.300072 run-command.c:655       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/gentoo-mirror/gentoo.git
16:37:04.301179 git.c:750               trace: exec: git-remote-https origin https://github.com/gentoo-mirror/gentoo.git
16:37:04.301206 run-command.c:655       trace: run_command: git-remote-https origin https://github.com/gentoo-mirror/gentoo.git
POST git-upload-pack (416 bytes)
POST git-upload-pack (260 bytes)
remote: Enumerating objects: 3932879, done.
remote: Counting objects: 100% (3932848/3932848), done.
remote: Compressing objects: 100% (1250804/1250804), done.
16:41:51.365577 run-command.c:655       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 371837 on laptop' --pack_header=2,3899134
16:41:51.366911 git.c:460               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 371837 on laptop' --pack_header=2,3899134
remote: Total 3899134 (delta 2715016), reused 3824261 (delta 2645474), pack-reused 0
Receiving objects: 100% (3899134/3899134), 1.23 GiB | 3.11 MiB/s, done.
Resolving deltas: 100% (2715016/2715016), completed with 16307 local objects.
16:49:08.902843 run-command.c:655       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
16:49:08.903936 git.c:460               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
From https://github.com/gentoo-mirror/gentoo
   c2bc8bc616e..bf7e240d597  stable     -> origin/stable
16:49:49.911466 run-command.c:1576      run_processes_parallel: preparing to run up to 1 tasks
16:49:49.912657 run-command.c:1614      run_processes_parallel: done
16:49:49.912667 run-command.c:655       trace: run_command: git maintenance run --auto --no-quiet
16:49:49.956276 git.c:460               trace: built-in: git maintenance run --auto --no-quiet

real    12m45.722s
user    2m44.294s
sys     0m20.641s
Comment 23 Larry the Git Cow gentoo-dev 2022-12-21 01:28:09 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ac405c994a0cce67e11c1bc7fb37fc02bba627ae

commit ac405c994a0cce67e11c1bc7fb37fc02bba627ae
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-12-17 05:19:32 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-12-21 01:28:03 +0000

    sync: git: allow truncating history if repository is marked volatile
    
    See 54a08c9f71dc37851fdb0364576581ca19df6580 for an explanation
    of the context here.
    
    If the repository is marked as volatile, it's assumed that
    Portage has complete control of it, and for Portage's purposes,
    there's no use in having extended history (in fact, if anything,
    it's deterimental, as it makes a sync more likely to fail due
    to the large disk space requirements over time).
    
    Bug: https://bugs.gentoo.org/824782
    Bug: https://bugs.gentoo.org/887025
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/sync/modules/git/git.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=da8e7d193563a2b0c88a523083b2a3466ff4526e

commit da8e7d193563a2b0c88a523083b2a3466ff4526e
Author:     Florian Schmaus <flow@gentoo.org>
AuthorDate: 2022-12-14 16:09:50 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-12-21 01:28:02 +0000

    sync: git: only perform shallow updates if the repository is shallow
    
    With the new default of shallow cloning/updating git repositories,
    added with f2207e41792d ("GitSync.update: default to "--depth 1" (bug
    824782 comment 17)"), existing *non-shallow* repositories would become
    shallow after a "emerge --sync".
    
    This adds a check to see if the repository is already shallow, and
    only in this case performs a shallow clone.
    
    Thanks to Douglas Freed for pointing out that this should consider
    whether or not the user explicitly set sync-depth.
    
    Thanks-to: Douglas Freed <dwfreed@mtu.edu>
    Bug: https://bugs.gentoo.org/824782
    Bug: https://bugs.gentoo.org/887025
    Closes: https://github.com/gentoo/portage/pull/960
    Signed-off-by: Sam James <sam@gentoo.org>

 NEWS                                |  2 ++
 lib/portage/sync/modules/git/git.py | 28 +++++++++++++++++++++++-----
 2 files changed, 25 insertions(+), 5 deletions(-)