Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 472020 - "eclean-pkg -n -d" keeps binpkgs with no corresponding ebuild
Summary: "eclean-pkg -n -d" keeps binpkgs with no corresponding ebuild
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords: InVCS, PATCH
Depends on:
Blocks: 346443
  Show dependency tree
 
Reported: 2013-06-01 16:24 UTC by Marc Joliet
Modified: 2016-06-23 22:17 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
modify eclean-pkg to also remove certain packages with no corresponding ebuild with the options "-d -n" (eclean_clean_nonexistent_binpkgs.patch,1.02 KB, patch)
2013-06-01 16:26 UTC, Marc Joliet
Details | Diff
Make "eclean-pkg -d -n" also remove packages with no corresponding ebuild (fix_package-names_with_destructive.patch,550 bytes, patch)
2014-11-30 14:14 UTC, Marc Joliet
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Marc Joliet 2013-06-01 16:24:50 UTC
Running "eclean-pkg -n -d" keeps binpkgs even if there is no corresponding ebuild, which is not what I expected from the documentation, which says:

  "Somewhere  in the middle, adding the --package-names option when using --destructive will protect files corresponding to all existing versions of installed packages."

I interpreted "existing versions of installed packages" to mean versions which are installed and which also exist in the portage tree.  However, what eclean does in findPackages() (with "-d -n") is three things:

- check whether the cat/pkg-ver is listed in an exclusion file
- check whether the cat/pkg-ver exists in the vartree, meaning it only filters out the newest version
- check whether the cat/pkg exists

The first point is irrelevant to this bug, and the second one is clear, since that's what "-d" by itself is supposed to do. However, the third point means that "eclean -d -n" will never delete binpkgs that belong to an installed cat/pkg.

In an attempt to fix this, I wrote a patch that changes findPackages() to also check that the package version exists, i.e., it changes the relevant conditional from

  if portage.cpv_getkey(cpv) in cp_all:

to

  if portage.cpv_getkey(cpv) in cp_all and port_dbapi.cpv_exists(cpv):

The patch also does two unrelated things (I hope you don't mind, I couldn't help myself):

- simplifies the code handling the package_names option by using a ternary operator (supported in all version of python in the tree, i.e., starting with 2.5)
- rather than put cp_all() into a dict (was this for performance maybe?) simply take the list returned by cp_all(), therefor all occurrences of cp_all are now lists rather than dicts

With this patch applied "eclean -d -n -p" reports almost 1 GB of packages to remove, including versions of samba and udev that don't exist anymore. The output appears to be the same as that of "eclean -p", save for a version of gentoo-sources that doesn't exist anymore, but that I still have installed (which I think verifies that the logic is correct).

My motivation for looking into this was that old firefox binpkgs were not being deleted, despite the fact that the corresponding ebuilds did not exist anymore. Furthermore, my weekly eclean cronjob kept reporting "Your packages directory was already clean.". In contrast, "eclean-dist -n -d" has been working fine so far and has, e.g., deleted all old firefox distfiles.


Reproducible: Always

Steps to Reproduce:
0. $PKGDIR contains packages with no corresponding ebuild, but of whose cat/pkg there is at least one installed version
1. execute "eclean-pkg -p"
2. execute "eclean-pkg -d -n -p"
Actual Results:  
"eclean-pkg -p" wants to remove obsolete packages, but "eclean-pkg -d -n -p" reports a clean $PKGDIR

Expected Results:  
"eclean-pkg -d -n -p" should remove packages belonging to an installed cat/pkg whose ebuilds don't exist anymore

$ emerge --info gentoolkit
Portage 2.1.11.62 (default/linux/amd64/13.0, gcc-4.6.3, glibc-2.15-r3, 3.8.13-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-3.8.13-gentoo-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4200+-with-gentoo-2.2
KiB Mem:     4046840 total,    848720 free
KiB Swap:    6291452 total,   6130976 free
Timestamp of tree: Fri, 31 May 2013 22:45:01 +0000
ld GNU ld (GNU Binutils) 2.22
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.3-r3, 3.2.3-r2
dev-util/cmake:           2.8.10.2-r2
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.9.6-r3, 1.11.6, 1.12.6
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.6.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo science wagnerflo sunrise overnight ladi proaudio pd-overlay mjoliet
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/polkit-1/actions"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --quiet-build=y --nospinner"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg collision-protect config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync metadata-transfer news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://de-mirror.org/distro/gentoo/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ ftp://mirror.muntinternet.net/pub/gentoo/"
LANG="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-s -j3"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS="xz"
PORTAGE_COMPRESS_FLAGS="-9"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/science /usr/local/portage/layman/wagnerflo /usr/local/portage/layman/sunrise /usr/local/portage/layman/overnight /usr/local/portage/layman/ladi /usr/local/portage/layman/pro-audio /usr/local/portage/layman/pd-overlay /usr/local/portage/marcec"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac acl alsa amd64 avahi bash-completion berkdb branding bzip2 cairo caps cdinstall cjk cli consolekit cracklib crypt css cups cxx dbus dga djvu dri dssi dts dvd encode exif fbcon ffmpeg fftw flac fortran ftp fuse gdbm gif gmp gnuplot gnutls gpm gtk iconv idn imlib inotify ipv6 jack jackmidi jpeg jpeg2k kipi ladspa lapack lash latex lcms libnotify libsamplerate logrotate lzma mad mjpeg mmx mmxext mng modplug modules mp3 mp4 mpeg mudflap multilib musepack musicbrainz ncurses nfs nls nptl ntp offensive ogg openexr opengl openmp osc pam pango pcre pdf plotutils png policykit pulseaudio qt4 quicktime rar readline rtsp samba sasl session sid slang smp sndfile speex spell sse sse2 sse3 ssl startup-notification svg sysfs taglib tcpd theora threads tiff timidity truetype unicode usb vaapi vdpau vim-syntax vorbis vpx webkit wma x264 xattr xcb xcomposite xface xft xml xmp xpm xscreensaver xv xvid zeroconf zlib zsh-completion" ABI_X86="64" ALSA_CARDS="ice1724 hda-intel usb-audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_US en en_GB de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" QEMU_SOFTMMU_TARGETS="x86_64" QEMU_USER_TARGETS="x86_64" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="radeon vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

app-portage/gentoolkit-0.3.0.7 was built with the following:
USE="(multilib)"
Comment 1 Marc Joliet 2013-06-01 16:26:53 UTC
Created attachment 349848 [details, diff]
modify eclean-pkg to also remove certain packages with no corresponding ebuild with the options "-d -n"
Comment 2 Zac Medico gentoo-dev 2013-06-02 00:41:09 UTC
Maybe it makes sense to add a new option for the behavior that you propose, since the --package-names description says that it protects _all_ versions.

(In reply to Marc Joliet from comment #0)
> - rather than put cp_all() into a dict (was this for performance maybe?)
> simply take the list returned by cp_all(), therefor all occurrences of
> cp_all are now lists rather than dicts

A dict uses a hash table for __contains__, while a list uses sequential search. Therefore, a dict is much more efficient for our use case.
Comment 3 Marc Joliet 2013-06-02 01:12:32 UTC
(In reply to Zac Medico from comment #2)
> Maybe it makes sense to add a new option for the behavior that you propose,
> since the --package-names description says that it protects _all_ versions.

That differs a bit from the man page, which says it protects *existing* versions (or perhaps I misunderstand "existing" in this case?).  Also, if that is the intended behaviour, then I have to wonder why eclean-dist *does* clean distfiles of non-existing ebuilds, since the man page says "-d -n" avoids recompiling and *downloading*, so if a binpkg is supposed to be protected, I would think the same should hold for the corresponding distfile.

Therefor it seems to me that eclean is inconsistent here. So if you think the additional option is the way to go, I think the behaviour of "--package-names" should still be modified.

Personally, I'm not so sure about adding an option for this.  The goal of the "-d -n" combination is to keep useful binpkgs/distfiles around without having $PKGDIR grow indefinitely (it's documented as being between "-d" and the default behaviour).  But the way it works now for binpkgs, $PKGDIR *does* grow indefinitely for as long as you have a package installed and new versions come along.  That's pretty bad for packages like wine, {libre,open}office and firefox.

Maybe the exclusion file is enough to handle the case of wanting/needing to keep the binpkg of a removed package version around in case an upgrade goes awry? But then again, you don't want to give the user an unpleasant surprise by removing any hope of having a downgrade path (espcially if you run eclean in a cron job, like I do).

> (In reply to Marc Joliet from comment #0)
> > - rather than put cp_all() into a dict (was this for performance maybe?)
> > simply take the list returned by cp_all(), therefor all occurrences of
> > cp_all are now lists rather than dicts
> 
> A dict uses a hash table for __contains__, while a list uses sequential
> search. Therefore, a dict is much more efficient for our use case.

Aha, I thought it might be something like that. Good to know!
Comment 4 Marc Joliet 2013-06-02 01:30:37 UTC
(In reply to Marc Joliet from comment #3)
[...]
> Maybe the exclusion file is enough to handle the case of wanting/needing to
> keep the binpkg of a removed package version around in case an upgrade goes
> awry? But then again, you don't want to give the user an unpleasant surprise
> by removing any hope of having a downgrade path (espcially if you run eclean
> in a cron job, like I do).

Or for this particular use case maybe better: the --time-limit and --interactive options.
Comment 5 Brian Dolbec gentoo-dev 2013-06-02 03:58:33 UTC
Thanks for that.  I've been at my daughters softball tournament this weekend.

When I did the eclean re-write. I actually did very little to the binpkg code.  It was already quite good and needed very little re-work to fit the new structure.  It was the distfiles portion of the code that was a complicated mess to sort out. and took the majority of the work.

I'll have a better look at it next week.  I haven't looked at that code in awhile.

Could be 4 more games in the semi & finals tomorrow if they keep winning to take first place :)

Now that my emaint rewrite is in portage.  I need to update eclean-pkg to use the api I created for it.
Comment 6 Marc Joliet 2013-07-01 12:14:20 UTC
(In reply to Brian Dolbec from comment #5)
[...]
> I'll have a better look at it next week.  I haven't looked at that code in
> awhile.
[...]

Have you had a chance to look into this yet?

In the meantime I've decided to patch it locally via /etc/portage/env/.
Comment 7 Marc Joliet 2014-11-30 14:14:44 UTC
Created attachment 390650 [details, diff]
Make "eclean-pkg -d -n" also remove packages with no corresponding ebuild

I remembered about this bug semi-recently and decided to simplify the patch following comment #2.  This version of the patch only extends the conditional as already discussed.  Comments?

(As with the original patch, I'm using this version on both of my systems.)
Comment 8 Paul Varner (RETIRED) gentoo-dev 2015-10-15 21:44:54 UTC
The patch has been pushed to the gentoolkit repository:

https://gitweb.gentoo.org/proj/gentoolkit.git/commit/?id=68283db9d0a775c334087a27601edf042987294c

It will be included in the next release.
Comment 9 Paul Varner (RETIRED) gentoo-dev 2016-06-23 22:17:27 UTC
This was released in gentoolkit-0.3.1