Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 123335 - change layout of distfiles
Summary: change layout of distfiles
Status: RESOLVED LATER
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Conceptual/Abstract Ideas (show other bugs)
Hardware: All Other
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-02-18 19:31 UTC by Michael Renner
Modified: 2006-03-06 06:02 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Renner 2006-02-18 19:31:42 UTC
This is the current state:

mirror:/storage/gentoo/data/source/distfiles# ls | wc -l
22359
mirror:/storage/gentoo/data/source/distfiles# cd ..
mirror:/storage/gentoo/data/source# ls -l | grep distfile
drwxr-xr-x   3 gentoo gentoo 950272 Feb 18 22:02 distfiles
mirror:/storage/gentoo/data/source#

People who want to browse the files "by hand" are usually stumped by the large output (the directory listing lighttpd creates is currently 4.2MB in size) and generation of those listings causes an excessive strain on the server.

Plus the creation/deletion of files doesn't scale too well on filesystems, which store directory entries in linked lists (ext2/3, probably the common bsd filesystems), since the list has to be traversed for each file deleted/created.

Introducing an additional directory hierarchy should fix this, and is the common solution for this problem for various projects, be it debian, cpan, slackware, etc.


One migration scenario for a better future:

Create subdirectories named after the first letter of each file and move the files in their respective directories. 

Either sym- or hardlink the files from the current distfiles root-directory to the specific directory where they reside in. (Check with the mirror admins first  (depending on the chosen linktype) if rsync hardlink support is enabled or their web/ftp servers allow/follow symlinks)

Adapt the build scripts so that they look for the files in their new location.

Change the scripts which fetch the files for distfiles so that they save them under the new location.

Wait a few weeks... (months? years? decades?) until the last user has updated and/or a clean upgrade-path exists, which doesn't rely on the old file locations.

Drop the sym/hardlinks.


After the change we'd have (with the current set of files) 63 subdirectories, the largest one containing 1767 files (letter 'g'), which is a definitive improvement over the current situation.

Full list can be seen at http://mirror.inode.at/gentoo-listing.txt .
Comment 1 SpanKY gentoo-dev 2006-02-18 21:56:27 UTC
this is something to be discussed on the gentoo-dev mailing list, not bugzilla