Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 741474 - =sys-apps/portage-3.0.4-r1 escapes rsync paths when fetching distfiles
Summary: =sys-apps/portage-3.0.4-r1 escapes rsync paths when fetching distfiles
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 719810
Blocks: 739718
  Show dependency tree
 
Reported: 2020-09-10 02:32 UTC by dkjii
Modified: 2020-11-22 07:22 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --fetchonly gtk+ (fetch.log,744 bytes, text/plain)
2020-09-10 02:32 UTC, dkjii
Details

Note You need to log in before you can comment on or make changes to this bug.
Description dkjii 2020-09-10 02:32:24 UTC
Created attachment 659346 [details]
emerge --fetchonly gtk+

>=sys-apps/portage-3.0.4-r1 escapes rsync paths when fetching distfiles. I had to downgrade to fetch gtk+ which was being escaped into gtk%2B.

portage-2.3.103-r1 does not escape the file path and works

reproduce:
/etc/portage/mirrors:
local rsync://srv/distfiles

emerge --fetchonly gtk+
see fetch.log

tested on sys-apps/portage-3.0.5

rsyncd from net-misc/rsync-3.2.3 doesn't recognize the escaping
Comment 1 Zac Medico gentoo-dev 2020-09-10 07:35:49 UTC
> >>> Fetching (1 of 1) x11-libs/gtk+-3.24.22::gentoo
> >>> Downloading 'rsync://srv.local/distfiles/distfiles/layout.conf'
> No digest file available and download failed.
> 
> !!! Couldn't download '.layout.conf.srv.local'. Aborting.
> >>> Downloading 'rsync://srv.local/distfiles/distfiles/gtk%2B-3.24.22.tar.xz'
> >>> Downloading 'http://distfiles.gentoo.org/distfiles/layout.conf'
> !!! Couldn't download '.layout.conf.distfiles.gentoo.org'. Aborting.
> >>> Downloading 'http://distfiles.gentoo.org/distfiles/gtk%2B-3.24.22.tar.xz'
> >>> Downloading 'https://download.gnome.org/sources/gtk+/3.24/gtk+-3.24.22.tar.xz'
> !!! Couldn't download 'gtk+-3.24.22.tar.xz'. Aborting.
>  * Fetch failed for 'x11-libs/gtk+-3.24.22'
> 
> >>> Failed to emerge x11-libs/gtk+-3.24.22

The actual file names on the mirrors are supposed to contain the escapes. For example, I've fetched the file just now:

> >>> Fetching (1 of 1) x11-libs/gtk+-3.24.22::gentoo
> >>> Downloading 'http://distfiles.gentoo.org/distfiles/layout.conf'
> --2020-09-10 00:30:51--  http://distfiles.gentoo.org/distfiles/layout.conf
> Resolving distfiles.gentoo.org... 2600:3402:200:227::2, 2600:3404:200:237::2, 2605:bc80:3010::134, ...
> Connecting to distfiles.gentoo.org|2600:3402:200:227::2|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 45 [text/plain]
> Saving to: ‘/var/cache/distfiles/.layout.conf.distfiles.gentoo.org.__download__’
> 
> /var/cache/distfiles/.layout.conf.distfiles.g 100%[=====================================================================================================>]      45  --.-KB/s    in 0s      
> 
> 2020-09-10 00:30:51 (1.78 MB/s) - ‘/var/cache/distfiles/.layout.conf.distfiles.gentoo.org.__download__’ saved [45/45]
> 
> >>> Downloading 'http://distfiles.gentoo.org/distfiles/94/gtk%2B-3.24.22.tar.xz'
> --2020-09-10 00:30:51--  http://distfiles.gentoo.org/distfiles/94/gtk%2B-3.24.22.tar.xz
> Resolving distfiles.gentoo.org... 2600:3402:200:227::2, 2600:3404:200:237::2, 2605:bc80:3010::134, ...
> Connecting to distfiles.gentoo.org|2600:3402:200:227::2|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 21349784 (20M) [application/x-xz]
> Saving to: ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’
> 
> /var/cache/distfiles/gtk+-3.24.22.tar.xz.__do 100%[=====================================================================================================>]  20.36M  4.40MB/s    in 5.5s    
> 
> 2020-09-10 00:30:57 (3.73 MB/s) - ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’ saved [21349784/21349784]
> 
>  * gtk+-3.24.22.tar.xz BLAKE2B SHA512 size ;-) ...
Comment 2 Zac Medico gentoo-dev 2020-09-10 07:41:53 UTC
It looks like maybe the file is saved with the wrong name on your server.
Comment 3 dkjii 2020-09-10 22:16:34 UTC
On the official mirror they are, but the file is saved in the distfiles folder as unescaped. This means you can't share your distfiles folder over rsync now for some files.

What I do with srv.local is I have a portage distfiles/ folder shared through rsyncd which acts as a cache to the official mirror.
Comment 4 dkjii 2020-09-10 22:17:38 UTC
(In reply to dkjii from comment #3)
> On the official mirror they are, but the file is saved in the distfiles
> folder as unescaped. This means you can't share your distfiles folder over
> rsync now for some files.
> 
> What I do with srv.local is I have a portage distfiles/ folder shared
> through rsyncd which acts as a cache to the official mirror.

you see it with the output of fetchonly:
>Saving to: ‘/var/cache/distfiles/gtk+-3.24.22.tar.xz.__download__’
Comment 5 Zac Medico gentoo-dev 2020-09-12 07:25:01 UTC
It looks like this quoting from bug 719810 needs to be conditional on protocol:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c238d5f7ed264179c263f5a2da983c4ee50b4f00
Comment 6 Zac Medico gentoo-dev 2020-09-12 20:03:00 UTC
For http/https protocol, the urlquote from bug 719810 is appropriate, since it protects file names that contain percent encoding like those shown in bug 719810 comment 0.

For rsync protocol, the urlquote is inappropriate, since rsync does not need any such protection and it interprets a percent encoded string literally.
Comment 7 Zac Medico gentoo-dev 2020-09-12 21:04:23 UTC
ftp uses percent encoding, like http and https
sftp does not use percent encoding, like rsync
Comment 9 Larry the Git Cow gentoo-dev 2020-09-14 07:23:42 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=20b6a36fbf791bcc7d42bd429cf8116399a014a7

commit 20b6a36fbf791bcc7d42bd429cf8116399a014a7
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-12 21:23:28 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-12 22:08:45 +0000

    get_mirror_url: urlquote only for ftp, http, and https (bug 741474)
    
    It's necessary to use urlquote for correct behavior with ftp, http,
    and https, since it's possible for file names to contain percent
    encoded characters that need to be protected by an additional layer
    of percent encoding. For other protocols such as sftp and rsync,
    all characters are interpreted literally, so urlquote must not be
    used.
    
    Fixes: c238d5f7ed264179c263f5a2da983c4ee50b4f00
    Bug: https://bugs.gentoo.org/719810
    Bug: https://bugs.gentoo.org/741474
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/package/ebuild/fetch.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
Comment 10 Larry the Git Cow gentoo-dev 2020-09-14 07:35:26 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1d7645a12b16a08ad45b1c82971d30e99e8ab589

commit 1d7645a12b16a08ad45b1c82971d30e99e8ab589
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-14 07:01:26 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-14 07:35:19 +0000

    sys-apps/portage: Bump to version 3.0.7
    
     #737470 egencache: add --external-cache-only option
     #741474 get_mirror_url: urlquote only for ftp, http, and https
     #742446 support riscv ilp32 and ilp32d ABIs
    
    Bug: https://bugs.gentoo.org/739718
    Bug: https://bugs.gentoo.org/737470
    Bug: https://bugs.gentoo.org/741474
    Bug: https://bugs.gentoo.org/742446
    Package-Manager: Portage-3.0.7, Repoman-3.0.1
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest             |   1 +
 sys-apps/portage/portage-3.0.7.ebuild | 267 ++++++++++++++++++++++++++++++++++
 2 files changed, 268 insertions(+)