Presently portage provides the -G and -g swtiches to allow precompiled binaries to be obtained and installed from a remote machine. However there are no tools to maintain such a selection of binaries. The first function builds a metadata.idx (looked for from within getbinpkg.py) from the contents of the binary directory. The advantage to this is that clients using -G or -g can download a prebuilt index, rather than doing it themselves, and it allows for the index to be accurate if packages are removed from the binary directory. Currently the code that builds the metadata file on the local machine only adds packages, it can't remove them from the index file and this can cause emerge to fail. The second function checks all of the binary packages against the current portage tree and advises if there are binary packages that are no longer in portage. Packages being dropped from portage is a problem for binary maintainers since the building of a new package is often problematic because there may be a number of dependencies that need upgrading. Since the emerge process stops each time an unmet dependency is found, it can be very slow going determining (and building) the packages that need updating as they are only apparent one at a time (e.g. portage only tells you of the immediate dependency, not subsequent ones, these can only be found after the first one is fixed). The advice about which packages have dropped out of portage means the repository can be maintained and regularly updated. If different versions of the same package are found the older ones can be moved to another directory so only the most current packages reside in the repository (this keeps the number of packages to a minimum and keep fixpackage run times down). SLOT's are respected here and only versions with the same SLOT are compared. Using this method updated packages can be added to the repository and the function called to clear out the older versions. These two functions are written in python and use functions and classes from portage.py, xpak.py and getbinpkg.py. The only site specific variables required for these functions are the location of the binary repository (and optionally a directory to place redundant binaries) on the filesystem. Theres still a bit of debug code in the functions, however they are in a working state (I use them here to maintain our binaries). Reproducible: Always Steps to Reproduce: N/A Actual Results: N/A Expected Results: N/A These functions contain debug code and could do with a bit of a clean up. If there is any interest in them, I'll sort this out.
Created attachment 21300 [details] python module containing two functions mentioned
Created attachment 21345 [details] Updated versions of previously supplied functions. Bug fix and improvement. User can now optionally be notified if there are updated ebuilds for binary packages. Removal of old/redundant packages modified, if a binary is marked as outdated and there is no other newer binary present, it isn't moved.
Created attachment 24787 [details] Wrapper script for binpkgmain.py Wrapper script for using binpkgmain.py. Put binpkgmain.py in /usr/lib/python2.2/site-packages (or elsewhere in the python library path) and invoke binpkgmain with --help for instructions on use. Remember that binpkgmain.py contains two variables (BIN_PKG_DIR and OLD_PKG_DIR) that need to be customised for the local machine.
There's a small problem with packages that live in different parts of the portage tree but share the same name. Here's an example: Traceback (most recent call last): File "binpkg.py", line 58, in ? out = binpkgmain.check_versions( move_red, show_upg ) File "/root/binpkgmain.py", line 73, in check_versions versions = portage.portdb.xmatch("match-visible", basename[1]) File "/usr/lib/portage/pym/portage.py", line 4681, in xmatch mydep=dep_expand(origdep,self) File "/usr/lib/portage/pym/portage.py", line 3258, in dep_expand return prefix+cpv_expand(mydep,mydb,use_cache=use_cache)+postfix File "/usr/lib/portage/pym/portage.py", line 3202, in cpv_expand raise ValueError, matches ValueError: ['app-emacs/w3', 'app-xemacs/w3'] I'm creating a common repository for about 5 machines that all are the same, portage-wise. So, I'd like to have all possible packages there so they only need to be compiled once. Thanks!
The error with identical named packages is (in my opinion) either a design flaw in portage, or an error made by the package maintainers. We have encountered this error with a number of packages, most specifically with involving clashes between xemacs and emacs packages. There is also an x11/eterm, emacs/eterm clash as well as dev-java/ant and vim/ant clash. I originally noticed this with eterm and at that time the bug reports and associated comments were of the opinion that no two packages should have the same name. In fact portage doesn't allow for this since /usr/portage/packages/All contains all binaries built for a system. The directory up from this contains the portage category directories and the contents are symlinks to /usr/portage/packages/All. So, by design, portage cannot cope with packages with the same name, since if two of them ever had an identical version number, one would over write the other in /usr/portage/packages/All. Maybe this is just an oversight and a result of binary packages not being the primary focus of gentoo.
Created attachment 30046 [details] Updated version of binpkgmain.py Changes: Added facility to have a file containing a list of packages for to be ignored by this module. This allows for peculiarities and bugs in portage where packages in different categories have the same name (e.g. x11-term/eterm and app-xemacs/eterm). Changed globals variables to reflect new server directory structure. Added functoins to check to see if redundant packages are dependancies of other remaining binaries before moving. A bit more clearer outpu required, but functional. Instantiated portsb class from portage correctly to get around the portage.portdb.xmatch hack I had been using. Replaced code that determined best version of a package out of a list with portdb.best method. Tidied up gen_metadata invocation to make suitable for use from wrapper script with user supplied binary directory. Currently only tested with portage2.0.49-r21
Created attachment 30047 [details] Updated wrapper script for binpkgmain.py Added new command line flag that allows for alternate binary directories to allow for machine specific groups of packages. If a number of packages a required only for a subset of machines, this allows those packages to be managed in a directory other than the global package directory.
I may have missed something, but I'm getting an error running the new scripts: # python binpkg.py Traceback (most recent call last): File "binpkg.py", line 4, in ? import binpkgmain File "/root/binpkgmain.py", line 14, in ? portdb = portage.portdbapi() TypeError: __init__() takes exactly 2 arguments (1 given)
What version of portage are you using?, currently I've only tested these scripts with 2.0.49-r21 (due to lack of time).
Good point, I'm current. 2.0.50-r6... I'll wait, since I can't help much :)
Related documentation about how to use these scripts and how to set up a central binary repository are now available at http://www.scms.waikato.ac.nz/~baz/
Created attachment 32046 [details] Updated version of binpkgmain.py - now handles full category/package names so there are no more clashes between the likes of app-emacs/apel and app-xemacs/apel. - Properly consults slot info and this is used all the way through. Previously this data was discarded midway through the script which has caused the odd problem (particularly the refusal to remove a redundant package even when a suitable replacement was in place, and the blank upgrade string seen when a package was removed from portage). - The annoying warning messages portage produced when starting the script have been suppressed. - Now works with latest portage version and python 2.3.3.
Created attachment 32115 [details] Updated wrapper script for binpkgmain.py Added ability to invoke binpkgmain with a: -t <pkgname> option which will check the metadata and indicate if this package is a dependancy of any others in the tree. Helps in determining if individual binaries can be moved out of the repository without breaking the build process.
Created attachment 32116 [details] Update version of binpkgmain.py Added function to support -t option in wrapper script
Much better so far. I'm running it with the -m flag and one item it dies on is this: Checking if gnome-base/gconf-2.4.0.1-r0 is needed by another package...Traceback (most recent call last): File "/root/binpkg.py", line 90, in ? out = binpkgmain.check_versions( move_red, show_upg ) File "/root/binpkgmain.py", line 272, in check_versions tbz2name = check_red_deps(t,p, metadata, red_ver[t][q]["best"]) File "/root/binpkgmain.py", line 398, in check_red_deps deps = string.split(metadata[z][y]) KeyError: 'DEPEND' gconf is still in portage today: Searching for package 'gconf' in all categories among: * installed packages * Portage tree (/usr/portage) * overlay tree (/usr/local/portage) [I--] [ ] gnome-base/gconf-2.6.0 (2) [-P-] [ ] gnome-base/gconf-1.0.8-r3 (1) [-P-] [ ] gnome-base/gconf-1.0.8-r5 (1) [-P-] [M~] gnome-base/gconf-2.6.1 (2) [-P-] [ ] gnome-base/gconf-2.4.0.1 (2) [-P-] [ ] gnome-base/gconf-1.0.9 (1) gconf-2.6.0.tbz2 exists if that's a consideration... I manually moved gconf-2.4.0.1.tbz2 to ../old_packages/ and ran it again. It dies on eel: Checking if gnome-base/eel-2.4.2-r0 is needed by another package...Traceback (most recent call last): File "/root/binpkg.py", line 90, in ? out = binpkgmain.check_versions( move_red, show_upg ) File "/root/binpkgmain.py", line 272, in check_versions tbz2name = check_red_deps(t,p, metadata, red_ver[t][q]["best"]) File "/root/binpkgmain.py", line 398, in check_red_deps deps = string.split(metadata[z][y]) KeyError: 'DEPEND' Eel to is in portage today: [-P-] [ ] gnome-base/eel-1.0.2-r3 (1) [-P-] [M~] gnome-base/eel-2.6.1 (2) [-P-] [ ] gnome-base/eel-2.6.0 (2) [-P-] [ ] gnome-base/eel-2.4.2 (2) Same routine for: gnome-base/libgtop-2.0.8-r0 gnome-extra/gnome-utils-2.4.1-r0 gnome-extra/nautilus-cd-burner-0.6.1-r0 dev-libs/libIDL-0.8.2-r0 gnome-extra/gconf-editor-2.4.0-r0 and a lot of others... Any thoughts on how I can help with this one?
Created attachment 32253 [details] Output of binpkg -m -d The attached output was produced and output this just as it died: Traceback (most recent call last): File "/root/binpkg.py", line 90, in ? out = binpkgmain.check_versions( move_red, show_upg ) File "/root/binpkgmain.py", line 272, in check_versions tbz2name = check_red_deps(t,p, metadata, red_ver[t][q]["best"]) File "/root/binpkgmain.py", line 398, in check_red_deps deps = string.split(metadata[z][y]) KeyError: 'DEPEND'
It looks like your metadata.idx file is corrupt. In the first instance try deleting it and recreating with binpkgmain -g. If that doesn't work, you'll need to enter a python shell (python at the command line) and type the following: import cPickle md = open("/path/to/metadata.idx", "r") m = cPickle.load(md) md.close m["gconf-2.6.0.tbz2"].keys() Send me the output from this, it should look something like: ['CATEGORY', 'PKGUSE', 'USE', 'DEPEND', 'LICENSE', 'gnome-utils-2.4.1.ebuild', 'PROVIDE', 'CXX', 'IUSE', 'RDEPEND', 'CXXFLAGS', 'CC', 'CDEPEND', 'CHOST', 'PDEPEND', 'CFLAGS', 'PF', 'CBUILD', 'environment.bz2', 'SLOT'] The error is occurring because the DEPEND key doesn't exist in the dictionary which is why I suspect the metadata.idx file is corrupt (and I should check the keys exist in the script - I'll do that for the next version).
Created attachment 32424 [details] Output of binpkg -m -d As suggested. Same problem results, even after rebuilding metadata.idx. This is the output from the set of commands you sent, run after rebuilding the .idx file: ['CATEGORY', 'PKGUSE', 'USE', 'DEPEND', 'LICENSE', 'PROVIDE', 'CXX', 'IUSE', 'RDEPEND', 'gconf-2.6.0.ebuild', 'CXXFLAGS', 'CC', 'CDEPEND', 'CHOST', 'PDEPEND', 'CFLAGS', 'PF', 'CBUILD', 'environment.bz2', 'SLOT']
Putting a hold on feature requests for portage as they are drowning out the bugs. Most of these features should be available in the next major version of portage. But for the time being, they are just drowning out the major bugs and delaying the next version's progress. Any bugs that contain patches and any bugs for etc-update or dispatch-conf can be reopened. Sorry, I'm just not good enough with bugzilla. ;)
Reopening for consideration.
We've had $PKGDIR/Packages index support for a long time now: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f45516f92d9cfa12736783ec62273e78f3805120 And there's `emaint --fix binhost` to fix it up after you remove packages (eclean-pkg calls it automatically).
