Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 877793 - FEATURES="force-mirror" causes wrong mirror url path to be used for some rust crates, thus 404-ing
Summary: FEATURES="force-mirror" causes wrong mirror url path to be used for some rust...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: AMD64 Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, PullRequest
Depends on: 912676
Blocks:
  Show dependency tree
 
Reported: 2022-10-21 00:12 UTC by Emanuel Czirai
Modified: 2023-10-03 15:39 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---

Attachments
emerge --info, 2 outputs with and without 'force-mirror' feature (3files.tar.gz,13.65 KB, application/gzip)
2022-10-21 00:12 UTC, Emanuel Czirai 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Emanuel Czirai 2022-10-21 00:12:16 UTC 
Created attachment 824901 [details]
emerge --info, 2 outputs with and without 'force-mirror' feature

When I use 'force-mirror' feature (ie. FEATURES='force-mirror' in make.conf), the url that emerge uses to wget some (or all?) crates from the mirror(s) is different(wrong, gives 404) than when not using 'force-mirror' !

With 'force-mirror', it tries this and it's a 404 not found:
https://distfiles.gentoo.org/distfiles/discard-1.0.4.crate

Without 'force-mirror', it tries the following and it's correct (found):
https://distfiles.gentoo.org/distfiles/86/discard-1.0.4.crate

also:

With 'force-mirror', you may notice something new(compared to without):
Warning: No mirrors available for file '.layout.conf.distfiles.gentoo.org'

But without 'force-mirror' it downloads that layout just fine:

>>> Downloading 'https://distfiles.gentoo.org/distfiles/layout.conf'
--2022-10-21 01:39:16--  https://distfiles.gentoo.org/distfiles/layout.conf
Connecting to distfiles.gentoo.org|156.146.33.18|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [text/plain]
Saving to: '/var/cache/distfiles/.layout.conf.distfiles.gentoo.org'

100%[================================================================================>]      45  --.-KB/s    in 0s      

2022-10-21 01:39:16 (17.2 MB/s) - '/var/cache/distfiles/.layout.conf.distfiles.gentoo.org' saved [45/45]

I'm not sure whether or not this layout being gotten(or not) influences the correctness of the path (eg. for `discard-1.0.4.crate`).


(it seems I'm limited to attaching only one file, but I need to attach 3, yet the attach message says `(without tar)`)
Comment 1 Emanuel Czirai 2022-10-21 00:37:55 UTC 
Ok, now I can confirm that the layout file not being gotten at all ever(due to 'force-mirror' being used), is the reason why the wget url is wrong. Once the layout is gotten once, it's cached in .mirror-cache.json 

# cat /var/cache/distfiles/.layout.conf.distfiles.gentoo.org
[structure]
0=filename-hash BLAKE2B 8
1=flat

# mv /var/cache/distfiles/.layout.conf.distfiles.gentoo.org /tmp
# mv /var/cache/distfiles/.mirror-cache.json  /tmp
# rm /var/cache/distfiles/discard-1.0.4.crate
# FEATURES="force-mirror" GENTOO_MIRRORS='https://distfiles.gentoo.org' emerge -1v librsvg --fetchonly --ask=n

...snip...
 * difflib-0.4.0.crate BLAKE2B SHA512 size ;-) ...                                                                                                         [ ok ]
Warning: No mirrors available for file '.layout.conf.distfiles.gentoo.org'
>>> Downloading 'https://distfiles.gentoo.org/distfiles/discard-1.0.4.crate'
--2022-10-21 02:35:13--  https://distfiles.gentoo.org/distfiles/discard-1.0.4.crate
Resolving distfiles.gentoo.org... wget: !! FUNCTION_NAME == 'gethostbyname' hostname='distfiles.gentoo.org'
wget: !! REENTRANT_NAME == 'gethostbyname_r' hostname='distfiles.gentoo.org'
212.102.56.179, 195.181.170.19, 212.102.56.143, ...
Connecting to distfiles.gentoo.org|212.102.56.179|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-10-21 02:35:13 ERROR 404: Not Found.

!!! Couldn't download 'discard-1.0.4.crate'. Aborting.
 * doc-comment-0.3.3.crate BLAKE2B SHA512 size ;-) ...                 
...snip...
 * yeslogic-fontconfig-sys-2.11.2.crate BLAKE2B SHA512 size ;-) ...                                                                                        [ ok ]
 * Fetch failed for 'gnome-base/librsvg-2.55.1'
Comment 2 Emanuel Czirai 2022-10-21 10:09:36 UTC 
Well, I'm happily using the following workaround, since I want FEATURES='force-mirror':

```patch
Index: /var/tmp/portage/sys-apps/portage-3.0.38.1/work/portage-3.0.38.1/lib/portage/package/ebuild/fetch.py
===================================================================
--- .orig/var/tmp/portage/sys-apps/portage-3.0.38.1/work/portage-3.0.38.1/lib/portage/package/ebuild/fetch.py
+++ /var/tmp/portage/sys-apps/portage-3.0.38.1/work/portage-3.0.38.1/lib/portage/package/ebuild/fetch.py
@@ -1094,9 +1094,10 @@ def fetch(
                 writemsg(_("Invalid mirror definition in SRC_URI:\n"), noiselevel=-1)
                 writemsg("  %s\n" % (myuri), noiselevel=-1)
         else:
-            if (restrict_fetch and not override_fetch) or force_mirror:
-                # Only fetch from specific mirrors is allowed.
-                continue
+            if myfile[:13] != '.layout.conf.':
+                if (restrict_fetch and not override_fetch) or force_mirror:
+                    # Only fetch from specific mirrors is allowed.
+                    continue
             primaryuris = primaryuri_dict.get(myfile)
             if primaryuris is None:
                 primaryuris = []
```
Comment 3 Emanuel Czirai 2022-10-30 21:32:00 UTC Comment hidden (obsolete)
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-10-30 21:33:38 UTC Comment hidden (obsolete)
Comment 5 Larry the Git Cow gentoo-dev 2023-09-17 20:29:16 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=542222ac998b655bcea7e491465e7df7f5e74591

commit 542222ac998b655bcea7e491465e7df7f5e74591
Author:     Mike Gilbert <floppym@gentoo.org>
AuthorDate: 2023-09-16 16:43:33 +0000
Commit:     Mike Gilbert <floppym@gentoo.org>
CommitDate: 2023-09-16 17:05:51 +0000

    fetch: ignore force-mirror feature when try_mirrors is false
    
    When fetching layout.conf, we call fetch(try_mirrors=False) to force
    direct use of the primary URI being passed and ignore mirrors.
    
    The force-mirror feature causes fetch to ignore the primary URI.
    
    Ignoring both the primary URI and mirrors leads to an unfetchable file.
    Give precedence to the try_mirrors parameter since we really only set it
    to False when fetching layout.conf from a mirror.
    
    Bug: https://bugs.gentoo.org/877793
    Signed-off-by: Mike Gilbert <floppym@gentoo.org>

 NEWS                                | 2 ++
 lib/portage/package/ebuild/fetch.py | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
Comment 6 Larry the Git Cow gentoo-dev 2023-10-03 15:39:20 UTC 
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9f9211a35abef13079f93a96f57f3a96083c69a7

commit 9f9211a35abef13079f93a96f57f3a96083c69a7
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-03 15:38:28 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-03 15:38:41 +0000

    sys-apps/portage: add 3.0.52
    
    Closes: https://bugs.gentoo.org/704866
    Closes: https://bugs.gentoo.org/877793
    Closes: https://bugs.gentoo.org/889300
    Closes: https://bugs.gentoo.org/900224
    Closes: https://bugs.gentoo.org/912676
    Closes: https://bugs.gentoo.org/912808
    Closes: https://bugs.gentoo.org/913070
    Closes: https://bugs.gentoo.org/913103
    Closes: https://bugs.gentoo.org/914159
    Closes: https://bugs.gentoo.org/915054
    Closes: https://bugs.gentoo.org/915119
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.52.ebuild | 235 +++++++++++++++++++++++++++++++++
 2 files changed, 236 insertions(+)