Currently, portage starts fetching from the leftmost item in SRC_URI and moves right as necessary. This is an interesting feature as it allows specifying a backup mirror for a tarball in the ebuild. Doing so for an ebuild from the main tree is probably useless but is very useful for an overlay ebuild. This is going to be a vital need for the R overlay server for example. There is no coding to be done for portage as the feature already exists there. I'd be surprised if pkgcore and paludis behaved differently but I haven't tried. I would like this to be solidified in a future EAPI yo make sure this behavior doesn't disappear and can be counted on in the future. Thanks, Denis.
Did Portage always behave like this? What about other package managers? Maybe we could specify it retroactively?
It's not that simple due to mirror://, mirrors, local mirrors etc. I'm against specifying it because it will unnecessarily restrict what a package mangler is allowed to do.
(In reply to comment #1) > Did Portage always behave like this? No, it hasn't always been reliable, though it is reliable with current stable portage. For example, here are a couple of fixes: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=16f2a01e156f2742677e9950bcf931239790c753 http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commitdiff;h=960792b9fd0e11b16efdc0b538231288b8727862 > Maybe we could specify it retroactively? Are we mostly just talking about RESTRICT=primaryuri here? Do you want to specify handling of thirdpartymirrors in this case?
(In reply to comment #2) > It's not that simple due to mirror://, mirrors, local mirrors etc. If Zac can do it you can certainly do it too. > it will unnecessarily restrict what a package > mangler is allowed to do. You're using the wrong end of your binoculars. It enhances the package mangler's feature set. All the rest is about how good you are at implementing a spec, and we all know you are more than good enough. Now, if you are concerned about not being able to fetch items in parallel, that is a valid concern. We could always decide that URIs of files with a different digest can be downloaded in parallel, while URIs of files with the same digest must be tried in the order they appear. This would give the best of both worlds, i.e. let you parallelize downloads while still allowing mirroring. Denis.
(In reply to comment #4) > You're using the wrong end of your binoculars. It enhances the package > mangler's feature set. All the rest is about how good you are at > implementing a spec, and we all know you are more than good enough. It means we get into a horrible mess specifying interactions with 'local' mirrors, background fetching, distfiles being shared between multiple packages and the like. We have three options: We could totally locks down exactly how things are fetched. This would disallow useful things, like packages sharing distfiles, and local mirrors, and background fetching. This wouldn't solve your problem, though, because such a spec wouldn't be enforceable. We could stick in some half-arsed restrictions. These would disallow useful behaviour, and still not provide whatever guarantees it is you think you're looking for. Or we could leave it as-is, work out exactly what problem we're trying to solve here, and then come up with a solution to that.
(In reply to comment #5) > disallow useful things, like packages sharing distfiles, and local mirrors, > and background fetching. I know I'm clueless compared to you, but I fail to see why what we're talking about interferes with this list of useful things. Especially knowing that portage already does it for quite a while. If there were any problem related to that we would surely know about it. > Or we could leave it as-is, work out exactly what problem we're trying to > solve here, and then come up with a solution to that. The fact is that there is no problem. Portage does what is needed already, so there is nothing to fix. Sure, paludis may not do it but it can't be hard to implement. I understand it must be painful for you to not see paludis as the gold standard of package managers in Gentoo. But let's face it, blocking progress in a tool that a majority uses due to past decisions in a tool that only a minority uses, however good it is, would be very wrong. Now, if you have suggestions on how to improve on this already existing feature I believe we'll all be happy to hear them. Denis.
You've been looking at this the wrong way: where possible, it's better to specify things that a package can rely upon, not how things are done. If there's something you'd like to be able to rely upon, state what it is. (If the thing you want to rely upon is "URLs to the left are fetched before those to the right", that's already not the case. Think of two packages with common items in SRC_URI in different orders.) This isn't about Paludis. Paludis probably already does what you want it to do. It's about coming up with a spec that makes sense. "Fetching from the leftmost" isn't a consistent requirement.
Looks like we're making progress. (In reply to comment #7) > You've been looking at this the wrong way: where possible, it's better to > specify things that a package can rely upon, not how things are done. If > there's something you'd like to be able to rely upon, state what it is. I said it in my first message: what portage does today (and based on what you wrote above maybe paludis too). It lets ebuild writers provide an alternative download location for a file, used only when the primary location doesn't work. Note that it has already been used by others than me, so there obviously is a need. Later I offered this possible improvement: > Now, if you are concerned about not being able to fetch items in parallel, > that is a valid concern. We could always decide that URIs of files with a > different digest can be downloaded in parallel, while URIs of files with the > same digest must be tried in the order they appear. This would give the best > of both worlds, i.e. let you parallelize downloads while still allowing > mirroring. Based on what I understand of what goes on there, I think that should be enough to make you and others happy. Feel free to correct me. (In reply to comment #7) > It's about coming up with a spec that makes sense. "Fetching from the > leftmost" isn't a consistent requirement. Agreed about a spec that makes sense. And if the wording doesn't suit you, I'm not attached to it. I don't want to tattle but that's how Zac described it to me. ;o) Note that I'm not attached to the syntax either. It could come in a completely different form that it wouldn't bother me. Denis.
Don't think about what Portage does (and it doesn't do what you claim except in trivial cases). Specify what you want to be able to rely upon. What happens when you've got this in one package: SRC_URI="http://first/blah http://second/blah" and this in another: SRC_URI="http://second/blah http://first/blah" What should happen then? And the bigger question, which seems to the the root of the problem: why are ebuilds supposed to care where their distfiles come from? Currently ebuilds don't have any way of knowing how a distfile was obtained, and so the method used to do downloading has no effect upon an ebuild's behaviour.
Alright. I'm going to explain a third time and go along with your questions below. Everybody else I talked to immediately understood what I meant. (In reply to comment #9) > Don't think about what Portage does Let's agree on one thing: you think of whatever you want, I think of whatever I want. Fair enough? > and it doesn't do what you claim except in trivial cases And that's all I'm after. Specifying a trivial case which is already present in portage and, by your own admission, paludis. You're the one who's trying to make that needlessly complicated. > What happens when you've got this in one package: > SRC_URI="http://first/blah http://second/blah" > and this in another: > SRC_URI="http://second/blah http://first/blah" > What should happen then? First case: try fetching from http://first/blah and if that doesn't work then try http://second/blah. Second case: the other way around. How difficult was that? Everybody else understood that immediately. You're making me doubt: are you the real Ciaran? > And the bigger question, which seems to the the root of the problem: why are > ebuilds supposed to care where their distfiles come from? I never said ebuilds had to care where their distfiles come from. I never thought ebuilds had to care where their distfiles come from. I do not see a reason why ebuilds would need to care where their distfiles come from. What the fetcher does based on input from ebuilds is none of the ebuilds' business. I suggest you stop playing dumb because I'm naturally gifted in that area and you stand no chance against me. All I'm after is that we specify a behavior which already exists to make sure it does not disappear. There is nothing to do except adding a few lines to PMS. If you disagree with that, fine, it's your right. But one dissentive opinion, however loud and repetitive it is, does not make a majority. Others could fall in your trap but I've seen you playing that game way too many times. If you have real, constructive, suggestions we'll be happy to know about them. But if you need to be explained yet another time what I meant above, I propose we do that in private as others are certainly bored enough with it already. Denis.
(In reply to comment #10) > Alright. I'm going to explain a third time and go along with your questions > below. Everybody else I talked to immediately understood what I meant. Everyone understands what you mean. The problem is that "what you mean" isn't something we can specify, for the two reasons I've already given. First, "what you mean" isn't logically consistent, and doesn't make sense. Second, "what you mean" is to specify a behaviour, rather than invariants, which we couldn't do without totally locking down the exact way in which fetching is done (thus ruling out parallel fetches, etc etc). > (In reply to comment #9) > > Don't think about what Portage does > > Let's agree on one thing: you think of whatever you want, I think of > whatever I want. Fair enough? No. We're trying to come up with a specification here. To do that, we need to think in terms of invariants. You need to tell us *what you want ebuilds to be able to rely upon*. And here's the thing: the answer to this isn't "things to the left are fetched before things to the right", because that already isn't what happens. > > and it doesn't do what you claim except in trivial cases > > And that's all I'm after. Specifying a trivial case which is already present > in portage and, by your own admission, paludis. You're the one who's trying > to make that needlessly complicated. All we'd be able to specify here is "things to the left might be fetched before things to the right, but they don't have to be". And that's useless: it doesn't allow ebuilds to rely upon anything. > First case: try fetching from http://first/blah and if that doesn't work > then try http://second/blah. > Second case: the other way around. ...and if both of those ebuilds are in the same resolution set? And if both are being installed in parallel? What happens then? > > And the bigger question, which seems to the the root of the problem: why are > > ebuilds supposed to care where their distfiles come from? > > I never said ebuilds had to care where their distfiles come from. I never > thought ebuilds had to care where their distfiles come from. I do not see a > reason why ebuilds would need to care where their distfiles come from. What > the fetcher does based on input from ebuilds is none of the ebuilds' > business. Then why do you want us to specify it? What are you trying to achieve here? You've still not told us what the actual use case for this is. > I suggest you stop playing dumb because I'm naturally gifted in that area > and you stand no chance against me. All I'm after is that we specify a > behavior which already exists to make sure it does not disappear. Ok. The current behaviour is "things to the left are sometimes fetched before things to the right, but sometimes they aren't, and there are no guarantees whatsoever". If I rewrite that formally, would that satisfy you? > There is nothing to do except adding a few lines to PMS. Fine. Please tell us what those few lines are. You don't even have to write it out formally. Just say what it is you want specified.
Is there still any interest in this? It seems that no progress have been made so far, and no new comment for 1.5yr.
(In reply to Michał Górny from comment #12) > Is there still any interest in this? It seems that no progress have been > made so far, and no new comment for 1.5yr. No reply for 3 years. Closing.
