I think it is far past time required for supporting ftp:// URLs natively in PM implementations. From glancing through the tree, many ebuilds and mirrors still using them have http(s) alternatives that should be used instead or relate to semi-dead projects that often rely on Gentoo mirroring anyway. Dropping support would also simplify workarounds in tooling such as pkgcheck that have had to add specialized support for ftp handling.
(In reply to Tim Harder from comment #0) > Dropping support would also simplify workarounds in tooling such as pkgcheck > that have had to add specialized support for ftp handling. Obviously this would be far down the road for most tools once support for older EAPIs has been dropped. However, I would suggest that most of the main tree should be migrated to available http(s) URL alternatives anyway before enforcing any bans via EAPI.
(In reply to Tim Harder from comment #1) > Obviously this would be far down the road for most tools once support for > older EAPIs has been dropped. So far, we have required package managers to keep support for previous EAPIs indefinitely. For example, such packages could still be installed on users' systems, so their EAPI would have to be known in order to uninstall them. I guess it is a separate discussion whether we want to limit the time for such support.
Also, note that PMS doesn't have any requirements for HOMEPAGE. It is basically opaque, and I think the only requirement is that the URIs have valid syntax. So one may even put "gopher:" or a URN like "doi:" there. AFAICS the PM won't do anything with the URIs in HOMEPAGE.
(In reply to Ulrich Müller from comment #2) > (In reply to Tim Harder from comment #1) > > Obviously this would be far down the road for most tools once support for > > older EAPIs has been dropped. > > So far, we have required package managers to keep support for previous EAPIs > indefinitely. For example, such packages could still be installed on users' > systems, so their EAPI would have to be known in order to uninstall them. > > I guess it is a separate discussion whether we want to limit the time for > such support. Personally I'd suggest this should be discussed in another bug and a support timeline be added to PMS or somewhere else semi-official. Forcing new implementations to support all EAPIs isn't a winning play and maintainers for older implementations probably could favor being able to drop legacy code at some point.
(In reply to Ulrich Müller from comment #3) > Also, note that PMS doesn't have any requirements for HOMEPAGE. It is > basically opaque, and I think the only requirement is that the URIs have > valid syntax. > > So one may even put "gopher:" or a URN like "doi:" there. AFAICS the PM > won't do anything with the URIs in HOMEPAGE. This feels semi-odd as clarifying it allows more strict checking of validity, e.g. pkgcheck checks http(s) links as well as more hackily checking ftp as well. Is opaque support actually valuable or more of a legacy choice that hasn't been questioned?
Would it be possible to get this accepted as an EAPI9 feature for at least SRC_URI? Distros like debian starting dropping ftp support nearly 8 years ago by now (https://lists.debian.org/debian-announce/2017/msg00001.html) and practically any active, valid host offers the same files via http(s) (even though some don't publicly announce that support) for SRC_URI usage while files only available via dead FTP hosts should be pointed somewhere else anyway.
Still too early IMHO. First we should try to reduce usage in the tree, and see how that goes. A quick scan of the tree shows that there are still packages with ftp-only SRC_URI. Also, no apparent alternatives for some of the upstreams.
(In reply to Ulrich Müller from comment #7) > Still too early IMHO. First we should try to reduce usage in the tree, and > see how that goes. I doubt that's going to happen much since pkgcheck doesn't warn about them currently AFAIK (while pkgcruft will flag them via invalid protocol results for UriInvalid reports). > A quick scan of the tree shows that there are still packages with ftp-only > SRC_URI. Also, no apparent alternatives for some of the upstreams. Some examples you found could be helpful. For example bash (and all its patches) could use https://ftp.gnu.org/gnu/bash/*, readline (and all its patches) could use https://ftp.gnu.org/pub/gnu/readline/, file could use http://ftp.astron.com/pub/file/*, postfix has tons of http(s) mirrors (with probably much higher bandwidth) but for some reason chooses to use ftp, mutt should use http://ftp.mutt.org/pub/mutt/ which it actually lists as primary, etc. Those were either the main offenders or programs I've used in the past that jumped out to me from a full tree scan. Furthermore, some packages list ftp fallbacks in SRC_URI while their upstream doesn't advertise FTP existence anymore, e.g. stunnel, sudo, etc. This leads me to think maintainers over the past decade may often not even notice if the FTP links work anymore especially in cases where the http(s) link is first in SRC_URI.
Also, just to note I'm aware `pkgcheck scan --net` should flag dead ftp links via DeadUrl results (since I added that myself), but networking support isn't enabled by default and I imagine DeadUrl results are still warnings so they're easier to ignore, especially because the default keywords list for --exit won't return a failure for them.
(In reply to Tim Harder from comment #8) > (In reply to Ulrich Müller from comment #7) > > Still too early IMHO. First we should try to reduce usage in the tree, and > > see how that goes. > > I doubt that's going to happen much since pkgcheck doesn't warn about them > currently AFAIK (while pkgcruft will flag them via invalid protocol results > for UriInvalid reports). > > > A quick scan of the tree shows that there are still packages with ftp-only > > SRC_URI. Also, no apparent alternatives for some of the upstreams. > > Some examples you found could be helpful. > > For example bash (and all its patches) could use > https://ftp.gnu.org/gnu/bash/*, readline (and all its patches) could use > https://ftp.gnu.org/pub/gnu/readline/, file could use For bash/readline, Chet often uploads to his own one first, and sometimes IIRC with quite a delay. I'll look at the other bits you mention though.
(In reply to Sam James from comment #10) > For bash/readline, Chet often uploads to his own one first, and sometimes > IIRC with quite a delay. Even for releases? I'm not overly surprised, but that still feels wrong. I will also note many (most?) other distributions (I checked arch, debian, and fedora) appear to entirely pull from the GNU mirrors so I'm not sure why we need to be special.
The issue is the patches he drops every so often, I think.
(In reply to Sam James from comment #12) > The issue is the patches he drops every so often, I think. Assuming the timestamps are correct (and relate to when the file was uploaded to the mirror), looking at the values between the patch commits at https://git.savannah.gnu.org/cgit/bash.git/log/ and the patch files on the GNU mirror at https://ftp.gnu.org/gnu/bash/bash-5.2-patches/ there doesn't appear to be much lag time recently, but I'll take your word for it since I don't package bash.
(In reply to Tim Harder from comment #13) > Assuming the timestamps are correct (and relate to when the file was > uploaded to the mirror), looking at the values between the patch commits at > https://git.savannah.gnu.org/cgit/bash.git/log/ and the patch files on the > GNU mirror at https://ftp.gnu.org/gnu/bash/bash-5.2-patches/ there doesn't > appear to be much lag time recently, but I'll take your word for it since I > don't package bash. And obviously it could be mentioned (but is generally implied for those familiar with upstream) that it would be nice if bash/readline/etc started using more recent development and release protocols so none of this would be an issue to begin with.
It's possible I'm either misremembering or perhaps more likely there's some delay in replication to ftp.gnu.org mirrors? dilfridge had to deal with such an issue when uploading a glibc release not long ago. But I'll try drop it and see how we go.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d17bfe0e0a4d5a30186b0db47b5f4a7fda04fdcf commit d17bfe0e0a4d5a30186b0db47b5f4a7fda04fdcf Author: Sam James <sam@gentoo.org> AuthorDate: 2025-02-23 00:55:12 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2025-02-23 01:32:02 +0000 net-misc/stunnel: update SRC_URI As pointed out by radhermit. No reason to use FTP for the archive URL and while at it, drop the other mirrors (I don't see why they need to be there as we have our own). Bug: https://bugs.gentoo.org/937512 Signed-off-by: Sam James <sam@gentoo.org> net-misc/stunnel/stunnel-5.68.ebuild | 10 ++-------- net-misc/stunnel/stunnel-5.71-r1.ebuild | 10 ++-------- net-misc/stunnel/stunnel-5.71.ebuild | 10 ++-------- 3 files changed, 6 insertions(+), 24 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cb607c8d859a5688e5cc6ad3b1df0249ed83755e commit cb607c8d859a5688e5cc6ad3b1df0249ed83755e Author: Sam James <sam@gentoo.org> AuthorDate: 2025-02-23 00:53:13 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2025-02-23 01:32:01 +0000 mail-client/mutt: update SRC_URI (ftp->http) Reported by radhermit. Bug: https://bugs.gentoo.org/937512 Signed-off-by: Sam James <sam@gentoo.org> mail-client/mutt/mutt-2.2.12-r1.ebuild | 4 ++-- mail-client/mutt/mutt-2.2.12.ebuild | 4 ++-- mail-client/mutt/mutt-2.2.13.ebuild | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c39a102fcacdd2f59fff06b085327c8529a6de99 commit c39a102fcacdd2f59fff06b085327c8529a6de99 Author: Sam James <sam@gentoo.org> AuthorDate: 2025-02-23 00:51:35 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2025-02-23 01:32:00 +0000 app-admin/sudo: update SRC_URI (drop ftp) Reported by radhermit. It was already secondary (well, "secondary", given SRC_URI doesn't actually have ordering) and upstream don't advertise it anymore. Bug: https://bugs.gentoo.org/937512 Signed-off-by: Sam James <sam@gentoo.org> app-admin/sudo/sudo-1.9.16_p2-r1.ebuild | 2 -- app-admin/sudo/sudo-9999.ebuild | 2 -- 2 files changed, 4 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a36e2db5c570f3862f4a86b9e35895b652a3525a commit a36e2db5c570f3862f4a86b9e35895b652a3525a Author: Sam James <sam@gentoo.org> AuthorDate: 2025-02-23 00:49:13 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2025-02-23 01:31:59 +0000 sys-apps/file: update SRC_URI (ftp->http) As reported by radhermit. Bug: https://bugs.gentoo.org/937512 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/file/file-5.45-r4.ebuild | 4 ++-- sys-apps/file/file-5.46-r2.ebuild | 4 ++-- sys-apps/file/file-9999.ebuild | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
(In reply to Tim Harder from comment #8) > (In reply to Ulrich Müller from comment #7) > > Still too early IMHO. First we should try to reduce usage in the tree, and > > see how that goes. > > I doubt that's going to happen much since pkgcheck doesn't warn about them > currently AFAIK (while pkgcruft will flag them via invalid protocol results > for UriInvalid reports). I was thinking about discussing this in the -dev ML: https://projects.gentoo.org/pms/8/pms.html#x1-80001.3 When reporting issues, remember that this document is not the appropriate place for pushing through changes to the tree or the package manager, except where those changes are bugs.
(In reply to Ulrich Müller from comment #17) > I was thinking about discussing this in the -dev ML: > > https://projects.gentoo.org/pms/8/pms.html#x1-80001.3 > When reporting issues, remember that this document is not the appropriate > place for pushing through changes to the tree or the package manager, except > where those changes are bugs. Not sure I agree with that, unless it implies undiscussed or personal changes. Otherwise, the entire spec is about pushing through changes to the tree. With respect to this change and others like it, devs will just keep copying outdated and unnecessary ftp usage until something stops them. I would hazard to guess there are a number of broken ftp links (I recall trying one that tarpitted incoming connections) in the tree right now for ancient projects that just keep getting copied on EAPI bumps since all the files are retrieved from the gentoo mirrors by default. Personally I think gentoo should have started migrating away after major browsers dropped internal ftp support (both chrome and firefox did so in 2021 iirc) as its semi-annoying nowadays to pull out an old-school ftp client just to browse an upstream directory.
(In reply to Tim Harder from comment #18) > > https://projects.gentoo.org/pms/8/pms.html#x1-80001.3 > > When reporting issues, remember that this document is not the appropriate > > place for pushing through changes to the tree or the package manager, except > > where those changes are bugs. > > Not sure I agree with that, unless it implies undiscussed or personal > changes. Otherwise, the entire spec is about pushing through changes to the > tree. This policy exists since 2007. There's also https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification/Future_EAPI_process "The change needs to be discussed on the gentoo-dev mailing list. For simple changes, the discussion as part of the eventual Council agenda item for the next EAPI may be enough. However, for more complex or controversial changes, starting a dedicated thread early is recommended." Related question, would this also affect profiles/thirdpartymirrors? The spec doesn't say anything about the URIs allowed in there. If we were to document Portage behaviour, then everything would be allowed when it is supported by FETCHCOMMAND. (Portage simply delegates the URI to FETCHCOMMAND without checking the protocol.)
(In reply to Ulrich Müller from comment #19) > This policy exists since 2007. There's also > https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification/ > Future_EAPI_process > "The change needs to be discussed on the gentoo-dev mailing list. For simple > changes, the discussion as part of the eventual Council agenda item for the > next EAPI may be enough. However, for more complex or controversial changes, > starting a dedicated thread early is recommended. Sure, that's what I meant by "undiscussed changes". Perhaps some of that wording and/or content from the wiki could be added to the spec to help clarify things. With respect to this change, yes it should be brought up on the dev list. I'd be interested in where people draw the line in terms of how many projects singly providing files via FTP is worth keeping support around for because I doubt it'll ever be zero as projects can often be semi-abandoned but still provide ancient tarballs over FTP (probably because the related infra hasn't been touched in over a decade either). > Related question, would this also affect profiles/thirdpartymirrors? The > spec doesn't say anything about the URIs allowed in there. If we were to > document Portage behaviour, then everything would be allowed when it is > supported by FETCHCOMMAND. (Portage simply delegates the URI to FETCHCOMMAND > without checking the protocol.) Yes, I think the thirdpartymirrors file would be included in this and as you note FETCHCOMMAND is a portage-ism that shouldn't be relied on. IMHO, it feels too vague to add to the spec as it currently is. At a quick glance, the only mirror with all FTP usage is "idsoftware" and that could be using https://www.gamers.org/pub/idgames/idstuff/ and/or the other geographic mirrors listed at https://www.gamers.org/pub/idgames/ assuming they're up and working (if we even need to have a mirror listing for such a small and old set of packages). I assume the other non-exclusive ftp mirrors in the thirdpartymirrors file could be dropped with no ill effects.
(In reply to Tim Harder from comment #20) > At a quick glance, the only mirror with all FTP usage is "idsoftware" and > that could be using https://www.gamers.org/pub/idgames/idstuff/ and/or the > other geographic mirrors listed at https://www.gamers.org/pub/idgames/ > assuming they're up and working (if we even need to have a mirror listing > for such a small and old set of packages). Alternatively the current FTP entries could be replaced by their HTTP alternatives: http://ftp.fu-berlin.de/pc/games/idgames/idstuff/ and https://www.gamers.org/pub/idgames/idstuff/ if we don't want to add new mirror entries. Often hosts for projects with semi-recently maintained infra will provide HTTP mirrors of FTP sites but forget to update their documentation to point to them as appears to be the case for the ftp.fu-berlin.de host.
To drive my point onward, the FTP server for freedesktop.org appears to be gone which currently both x11-libs/libGLw and net-misc/tigervnc refer to in SRC_URI. Honestly someone (preferably with write access to the tree so they can fix things) could script up a check to pass all pkgs using ftp links in SRC_URI and HOMEPAGE to something like `pkgcheck scan --net -k DeadUrl` just to get a feel for how many nonexistent ftp servers that gentoo still pretends exist. For example, here is a list of packages with failing ftp SRC_URI and/or HOMEPAGE links that I manually found while scanning categories `sys-*` and onwards with about a 75%+ failure rate of pkgs tested: sys-apps/isapnptools sys-apps/setserial sys-block/qla-fc-firmware sys-boot/sgibootcd sys-fs/mac-fdisk sys-fs/arm-fdisk www-apps/cvsweb x11-misc/xsetleds x11-misc/xsnap x11-plugins/gkrellfire Alternatively, infra could schedule a network check only scan with pkgcheck to run at a set time interval and merge the results into the qa reports in some fashion which might help get a feel for how many dead links are in the tree overall with the understanding that network access is flaky and the results won't always be 100% correct.
One funny thing here is, I can't actually check with Firefox whether the FTP sites remain or browse them to see what they serve, given the support was removed 4 years ago :)
(In reply to Sam James from comment #23) > One funny thing here is, I can't actually check with Firefox whether the FTP > sites remain or browse them to see what they serve, given the support was > removed 4 years ago :) Yep, that's what I mentioned being annoyed about in comment #18 and is part of why it would be nice to get this done sooner than later. :)
(In reply to Sam James from comment #23) > One funny thing here is, I can't actually check with Firefox whether the FTP > sites remain or browse them to see what they serve, given the support was > removed 4 years ago :) lynx, links and w3m still support FTP, and so do curl and wget.
(The point is I have to go out of my way and not use my usual method.)
(In reply to Ulrich Müller from comment #25) > lynx, links and w3m still support FTP, and so do curl and wget. If the point is just to see if the server exists that's basically what `pkgcheck scan --net -k DeadUrl` does in a more efficient fashion. IMHO, it isn't overly worthwhile to spend lots of time manually browsing ftp servers to see if maybe a file was moved if the server is still alive. Honestly, for every DeadUrl result from pkgcheck, I'd check if an http(s) mirror exists and if not update it to use the gentoo mirrors exclusively (or however you want to handle SRC_URI for dead upstreams).
