Currently, local distfiles occupy a flat namespace, usually under /usr/portage/disfiles. Portage assumes that the distfile's basename is a sufficiently unique identifier to distinguish it from all other distfiles. This is not always true. There are some cases where distfiles need to be renamed or occupy subdirectories. Case in point: I wrote an ebuild for PLT Scheme (which I already contributed). All versions of PLT are distributed in a file called plt.src.x.tar.gz, regardless of version. They are located in a directory on the server which indicates the version. So, an example URI to fetch from is "http://www.cs.utah.edu/plt/download/202/plt/plt.src.x.tar.gz" I just bumped the version of the PLT ebuild from 202 to 203, but portage can't distinguish the distfiles. This means that upgrading from one version of PLT to the next will never work if an old distfile is still there. Even worse, it appears that it will not work even if there isn't an old distfile in place because the 202 version of plt.src.x.tar.gz is already on the gentoo mirrors without any version information. Right now, when I try to emerge my new 203 version of the plt ebuild, portage (2.0.46-r12) is fetching the same old 202 version of plt.src.x.tar.gz from a Gentoo mirror even though I added "RESTRICT=nomirror" to the ebuild. It seems the only way I can get an upgrade to PLT to work is to manually place the correct version of of plt.src.x.tar.gz in the distfiles directory before emerging it. I can't even use "RESTRICT=fetch" and pkg_nofetch() trick to inform the user what to do. Unless I've missed something, the only way I can think of to solve this without changing Portage would be to append a version to distfiles on the Gentoo mirrors so they would be named something like plt.src.x-203.tar.gz. However, this would require changing the procedure for adding a distfile to the Gentoo mirrors and would not allow for downloading from the original sites. A real solution would allow an ebuild to specify the local name and location of each distfile. I would like to see the distfile directory structure reflect the main portage structure, but of course that wouldn't be necessary in most cases. It seems there are no ebuild functions that are called before the distfile digests are checked. Allowing the ebuild the override the fetch function would at least partly solve the problem. Reproducible: Always Steps to Reproduce:
Note the requirements in the submission of the ebuild, and a dev can put a properly named file on the master gentoo mirror for circulation.
I cannot find anything relevant in the "requirements in the submission of the ebuild." I looked in "The Developers HOWTO," "The Contributing Ebuilds Guide," and "Gentoo Linux Development Policy" and found nothing about naming distfiles or how to get distfiles onto mirrors. Since the ebuild I submitted for PLT Scheme points to the distribution tarball from the original site, a copy of that tarball is now on the Gentoo mirrors. The name of that file has no version information, so it will have to be changed to allow multiple versions of the ebuild. I see now way to influence how files get onto the distfiles mirrors. Perhaps I am mistaken and there is a procedure, but I have seen no reference to it.
Can we get this added to the docs?
I have this same problem with an ebuild that I wrote. The "rename it on ibiblio" isn't a solution since some packages restrict redistribution. I would suggest that /usr/portage/distfiles follows the same convention as packages so we have eg. /usr/portage/distfiles/dev-lang/package-x.y.z-r1/ as the base directory for files. Of course symlinks could be used for files that already exist (maybe symlink to a directory with unique filenames like "distfiles/hashes/5ef875.."). This suggestion should avoid any collisions in the distfile name space.
I'm going to take a look at this one. Mentioning this in the Ebuild HOWTO (gentoo-howto.xml) should be sufficient, no?
Mentioning what exactly? As far as I can see, this is a deficiency (I'm not sure if you can call it a bug) in Portage. It can't be fixed by additional documentation alone.
Ok, following the call :). Johnathan: Yes it can, if the ebuild is precessed correctly originally. The developer taking care of it should have done appropriate mangling of SRC_URI and also fetched and renamed source and put it up on distfiles-local, so that it gets mirrored properly. This is what should be added to docs, more precisely to the developer policy. Nicolas: I think the easiest way to go about this is to reassign the ebuild to docs-team unless Sven is a doc-developer, in which case Sven, could you please take care of that? If you need some details on what exactly has to be done, please drop me a note, I'll write up a paragraph for you. Actually, looking at the ebuild, I see that SRC_URI contains ${PV} in the path, but not in the name. In this case the situation could have been be resolved by just adding RESTRICT="nomirror". However I am afraid it's too late now :(. I am not that sure about how removing files on mirrors works. Johnathan: anyway, new version should have appropriately mangled SRC_URI and the renamed source shoud be put up for mirroring. As for the version in portage I guess we will have to stick with what's on place now.. Chris: this situation (restricted redistribution) seems quite a bit easier: user has to download it manually anyway, so you are going to have RESTRICT="fetch" in your ebuild and corresponding comment output anyway. Just add to that comment that user should also rename the fetched package to some acceptable form (perheaps the standard $PN-$PV is the easiest), and adjust SRC_URI corresponingly. This is only going to take one additional second of his life... George
I'm a doc-developer, I'll take a look but there was some discussion on #gentoo-dev that adding notes about all possible situations would make it more unreadable. Anyway: renaming the tarball and putting it in distfiles-local is a way to handle stuff, but it's better to mark it upstream and ask those developers to use versions in their tarballs. About the non-distributable tarball without version on it: if you find such an ebuild, bug it. That should be handled individually. I'll ask some more...
Has anyone contacted the upstream author to ask if they could provide the package releases with a versioned filename? Perhaps they could provide a directory that contains properly named symlinks to the actual tarballs elsewhere in the tree? This should always be the first step before coming up with workarounds that will require deviation of standard procedures. The fact that this package does not follow this common naming convention will likely cause similar inconveniences for others in the future.
I personally think that we need to look into providing more finegrained distfiles. The best proposal I heard up til now is name rewriting. An ebuild specifies how a distfile is to be called. Portage then needs to be smart enough to use the renamed file from the mirrors, but the normal one from the main repository.
With restricted redistribution the user doesn't have to manually download, it just means the files can't be renamed and mirrored on ibiblio, but can only be downloaded from the original web site ie. RESTRICT="nomirror". Surely there should be some way to handle this case and avoid namespace collisions at the same time? The proposal above for using subdirectories and hashes also has the advantage that you can easily see which packages are downloaded for each ebuild revision, and packages aren't needlessly downloaded between revisions.
It would be helpful to ask the original developer of the package to rename it to be easier to deal with, and I will do so. However, it's not reasonable or practical to expect the developer of any package which will potentially be in Portage to bend to its expectations of naming convention. In fact, it's kind of arrogant and self-centered (I'm not speaking of a particular person, just the attitude). Gentoo is not the only OS distribution. The namespace collision I encountered was a result of Portage's design, not a flaw in the original package. Perhaps in an ideal world, all tarballs would follow the same naming conventions. Why not add a single ebuild variable, say DISTFILE_SUBPATH, to specify the location of that ebuild's distfiles. It would default to "", placing distfiles at /usr/portage/distfiles, or wherever. So, in the case of PLT, I would set DISTFILE_SUBPATH=plt/202/ which would result in the file being downloaded to /usr/portage/distfiles/plt/202/plt.src.x.tar.gz. Details of the exact path would be left to the ebuild writer, and the great majority of ebuilds would require no change. I believe this would be simpler and have less potential for confusion than using the hashed names or renaming distfiles on mirrors. The only people who would have to be concerned with the details of distfile naming would be the original package developer and the ebuild developer.
I'm marking this as WONTFIX. There is currently no need to alter Portage to avoid these extremely few collisions. If you ever have such a collision problem, solve it as you like (asking the package developer to put up versions in it's tarball-path, or renaming and putting it on ibiblio). If you encounter something that cannot be fixed with one of the above solutions, file a seperate bug please. Do note that this "WONTFIX" can be (but this is no promise) a FIXLATER. However, I set WONTFIX just to be on the sure side. Current Portage development is aiming at making a stable, good working release for Gentoo 1.4. Afterwards more significant changes (such as special distfiles layouts) can be incorporated.
