Created attachment 659346 [details] emerge --fetchonly gtk+ >=sys-apps/portage-3.0.4-r1 escapes rsync paths when fetching distfiles. I had to downgrade to fetch gtk+ which was being escaped into gtk%2B. portage-2.3.103-r1 does not escape the file path and works reproduce: /etc/portage/mirrors: local rsync://srv/distfiles emerge --fetchonly gtk+ see fetch.log tested on sys-apps/portage-3.0.5 rsyncd from net-misc/rsync-3.2.3 doesn't recognize the escaping
> >>> Fetching (1 of 1) x11-libs/gtk+-3.24.22::gentoo > >>> Downloading 'rsync://srv.local/distfiles/distfiles/layout.conf' > No digest file available and download failed. > > !!! Couldn't download '.layout.conf.srv.local'. Aborting. > >>> Downloading 'rsync://srv.local/distfiles/distfiles/gtk%2B-3.24.22.tar.xz' > >>> Downloading 'http://distfiles.gentoo.org/distfiles/layout.conf' > !!! Couldn't download '.layout.conf.distfiles.gentoo.org'. Aborting. > >>> Downloading 'http://distfiles.gentoo.org/distfiles/gtk%2B-3.24.22.tar.xz' > >>> Downloading 'https://download.gnome.org/sources/gtk+/3.24/gtk+-3.24.22.tar.xz' > !!! Couldn't download 'gtk+-3.24.22.tar.xz'. Aborting. > * Fetch failed for 'x11-libs/gtk+-3.24.22' > > >>> Failed to emerge x11-libs/gtk+-3.24.22 The actual file names on the mirrors are supposed to contain the escapes. For example, I've fetched the file just now: > >>> Fetching (1 of 1) x11-libs/gtk+-3.24.22::gentoo > >>> Downloading 'http://distfiles.gentoo.org/distfiles/layout.conf' > --2020-09-10 00:30:51-- http://distfiles.gentoo.org/distfiles/layout.conf > Resolving distfiles.gentoo.org... 2600:3402:200:227::2, 2600:3404:200:237::2, 2605:bc80:3010::134, ... > Connecting to distfiles.gentoo.org|2600:3402:200:227::2|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 45 [text/plain] > Saving to: ‘/var/cache/distfiles/.layout.conf.distfiles.gentoo.org.__download__’ > > /var/cache/distfiles/.layout.conf.distfiles.g 100%[=====================================================================================================>] 45 --.-KB/s in 0s > > 2020-09-10 00:30:51 (1.78 MB/s) - ‘/var/cache/distfiles/.layout.conf.distfiles.gentoo.org.__download__’ saved [45/45] > > >>> Downloading 'http://distfiles.gentoo.org/distfiles/94/gtk%2B-3.24.22.tar.xz' > --2020-09-10 00:30:51-- http://distfiles.gentoo.org/distfiles/94/gtk%2B-3.24.22.tar.xz > Resolving distfiles.gentoo.org... 2600:3402:200:227::2, 2600:3404:200:237::2, 2605:bc80:3010::134, ... > Connecting to distfiles.gentoo.org|2600:3402:200:227::2|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 21349784 (20M) [application/x-xz] > Saving to: ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’ > > /var/cache/distfiles/gtk+-3.24.22.tar.xz.__do 100%[=====================================================================================================>] 20.36M 4.40MB/s in 5.5s > > 2020-09-10 00:30:57 (3.73 MB/s) - ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’ saved [21349784/21349784] > > * gtk+-3.24.22.tar.xz BLAKE2B SHA512 size ;-) ...
It looks like maybe the file is saved with the wrong name on your server.
On the official mirror they are, but the file is saved in the distfiles folder as unescaped. This means you can't share your distfiles folder over rsync now for some files. What I do with srv.local is I have a portage distfiles/ folder shared through rsyncd which acts as a cache to the official mirror.
(In reply to dkjii from comment #3) > On the official mirror they are, but the file is saved in the distfiles > folder as unescaped. This means you can't share your distfiles folder over > rsync now for some files. > > What I do with srv.local is I have a portage distfiles/ folder shared > through rsyncd which acts as a cache to the official mirror. you see it with the output of fetchonly: >Saving to: ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’
It looks like this quoting from bug 719810 needs to be conditional on protocol: https://gitweb.gentoo.org/proj/portage.git/commit/?id=c238d5f7ed264179c263f5a2da983c4ee50b4f00
For http/https protocol, the urlquote from bug 719810 is appropriate, since it protects file names that contain percent encoding like those shown in bug 719810 comment 0. For rsync protocol, the urlquote is inappropriate, since rsync does not need any such protection and it interprets a percent encoded string literally.
ftp uses percent encoding, like http and https sftp does not use percent encoding, like rsync
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/cf0dd906c8e3e929984f6750011b8035 https://github.com/gentoo/portage/pull/620
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=20b6a36fbf791bcc7d42bd429cf8116399a014a7 commit 20b6a36fbf791bcc7d42bd429cf8116399a014a7 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-12 21:23:28 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-12 22:08:45 +0000 get_mirror_url: urlquote only for ftp, http, and https (bug 741474) It's necessary to use urlquote for correct behavior with ftp, http, and https, since it's possible for file names to contain percent encoded characters that need to be protected by an additional layer of percent encoding. For other protocols such as sftp and rsync, all characters are interpreted literally, so urlquote must not be used. Fixes: c238d5f7ed264179c263f5a2da983c4ee50b4f00 Bug: https://bugs.gentoo.org/719810 Bug: https://bugs.gentoo.org/741474 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1d7645a12b16a08ad45b1c82971d30e99e8ab589 commit 1d7645a12b16a08ad45b1c82971d30e99e8ab589 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-14 07:01:26 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-14 07:35:19 +0000 sys-apps/portage: Bump to version 3.0.7 #737470 egencache: add --external-cache-only option #741474 get_mirror_url: urlquote only for ftp, http, and https #742446 support riscv ilp32 and ilp32d ABIs Bug: https://bugs.gentoo.org/739718 Bug: https://bugs.gentoo.org/737470 Bug: https://bugs.gentoo.org/741474 Bug: https://bugs.gentoo.org/742446 Package-Manager: Portage-3.0.7, Repoman-3.0.1 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.7.ebuild | 267 ++++++++++++++++++++++++++++++++++ 2 files changed, 268 insertions(+)